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To our families 


Preface 


This book has been designed for a final year undergraduate course in stochastic 
processes. It will also be suitable for mathematics undergraduates and others 
with interest in probability and stochastic processes, who wish to study on their 
own. | | 
The main prerequisite is probability theory: probability measures, random 

variables, expectation, independence, conditional probability, and the laws of 
large numbers. The only other prerequisite is calculus. This covers limits, series, 
the notion of continuity, differentiation and the Riemann integral. Familiarity 
with the Lebesgue integral would be a bonus. A certain level of fundamental 
mathematical experience, such as elementary set theory, is assumed implicitly. 

Throughout the book the exposition is interlaced with numerous exercises, 
which form an integral part of the course. Complete solutions are provided at 
the end of each chapter. Also, each exercise is accompanied by a hint to guide 
the reader in an informal manner. This feature will be particularly useful for 
self-study and may be of help in tutorials. It also presents a challenge for the 
lecturer to involve the students as active participants in the course. 

A brief introduction to probability is presented in the first chapter. This is 
mainly to fix terminology and notation, and to provide a survey of the results 
which will be required later on. However, conditional expectation is treated in 
detail in the second chapter, including exercises designed to develop the nec- 
essary skills and intuition. The reader is strongly encouraged to work through 
them prior to embarking on the rest of this course. This is because conditional 
expectation is a key tool for stochastic processes; which often presents some 
difficulty to the beginner. 

Chapter 3 is about martingales in discrete time. We study the basic prop- 
erties, but also some more advanced ones vai e STOPPING times and the Optional 
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Doob’s inequalities and convergence results. Chapter 5 is devoted to time- 
homogenous Markov chains with emphasis on their ergodic properties. Some 
important results are presented without proof, but with a lot of applications. 
However, Markov chains with a finite state space are treated in full detail. 
Chapter 6 deals with stochastic processes in continuous time. Much emphasis 
is put on two important examples, the Poisson and Wiener processes. Various 
properties of these are presented, including the behaviour of sample paths and 
the Doob maximal inequality. The last chapter is devoted to the It6 stochastic 
integral. This is carefully introduced and explained. We prove a stochastic ver- 
sion of the chain rule known as the Ito formula, and conclude with examples 
and the theory of stochastic differential equations. 

It is a pleasure to thank Andrew Carroll for his careful reading of the final 
draft of this book. His many comments and suggestions have been invaluable 
to us. We are also indebted to our students who took the Stochastic Analysis 
course at the University of Hull. Their feedback was instrumental in our choice 
of the topics covered and in adjusting the level of exercises to make them 
challenging yet accessible enough to final year undergraduates. 

As this book is going into its 3rd printing, we would like to thank our 
students and readers for their support and feedback. In particular, we wish 
to express our gratitude to Iaonnis Emmanouil of the University of Athens 
and to Brett T. Reynolds and Chris N. Reynolds of the University of Wales 
in Swansea for their extensive and meticulous lists of remarks and valuable 
suggestions, which helped us to improve the current version of Basic Stochastic 
Processes. 

We would greatly appreciate further feedback from our readers, who are 
invited to visit the Web Page http://www. hull.ac.uk/php/mastz/bsp. html 
for more information and to check the latest corrections in the book. 


Zdzistaw Brzezniak and Tomasz Zastawniak 
Kingston upon Hull, June 2000 
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Review of Probability 


In this chapter we shall recall some basic notions and facts from probability 
theory. Here is a short list of what needs to be reviewed: 


1) Probability spaces, o-fields and measures; 
2) Random variables and their distributions; 
3) Expectation and variance; 

4) The o-field generated by a random variable; 
5) Independence, conditional probability. 


The reader is advised to consult a book on probability for more information. 


1.1 Events and Probability 


Definition 1.1 


Let N be a non-empty set. A o-field F on Nisa family of subsets of N such 
that 


1) the empty set Ø belongs to F; 
2) if A belongs to F, then so does the complement 2 \ A; 
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3) if A1, Ag,... is a sequence of sets in F, then their union A; U A2 U--- also 
belongs to F. 


Example 1.1 


Throughout this course R will denote the set of real numbers. The family of 
Borel sets F = B(R) is a o-field on R. We recall that B(IR) is the smallest 
o-field containing all intervals in R. 


Definition 1.2 
Let F be a o-field on §2. A probability measure P is a function 
P : F => [0,1] 
such that 
1) P(Q) =1; 


2) if A,,Ao,... are pairwise disjoint sets (that is, A4; N A; = @ for i # 7) 
belonging to F, then 


PA UA = Pee e 


The triple (N, F, P) is called a probability space. The sets belonging to F 
are called events. An event A is said to occur almost surely (a.s.) whenever 
PCA). = I: 


Example 1.2 


We take the unit interval R = [0,1] with the o-field F = B((0,1]) of Borel 
sets B C [0,1], and Lebesgue measure P = Leb on [0,1]. Then (2,7, P) isa 
probability space. Recall that Leb is the unique measure defined on Borel sets 
such that 

Leb[a, b] = b—a 


for any interval [a,b]. (In fact Leb can be extended to a larger o-field, but we 


shall need Borel sets only.) 


Exercise 1.1 


Show that if A,,Ao,... is an expanding sequence of events, that is, 


A, CAgC:::, 
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then 
P(A, U Ap U---) = lim PAg) 
n—> co 
Similarly, if Ay, Ag,... is a contracting sequence of events, that is, 
Aj DAQgD:°::: , 
then 
P(A; NA2.M-:-) = lim P(A,). 
n> CO 
Hint Write Ai U A2 U--- as the union of a sequence of disjoint events: start with 


Ai, then add a disjoint set to obtain A; U A2, then add a disjoint set again to obtain 
A, U A2 U Az, and so on. Now that you have a sequence of disjoint sets, you can use 
the definition of a probability measure. To deal with the product A; ON A2M--- write 
it as a union of some events with the aid of De Morgan’s law. 


Lemma 1.1 (Borel—Cantelli) 
Let A,,A2,... be a sequence of events such that P(A,) + P(A2) +- < co 
and let Bn = An Aye? Us . Then P (B: NB2N---) ==); 


Exercise 1.2 


Prove the Borel—Cantelli lemma above. 


Hint Bı, Bo,... is a contracting sequence of events. 


1.2 Random Variables 


Definition 1.3 
If F is a o-field on 92, then a function £: 2 — R is said to be F-measurable if 
{EE BlLeF 


for every Borel set B € B(R). If (82,7, P) is a probability space, then such a 
function € is called a random variable. 


Remark 1.1 


A short-hand notation for events such as {€ € B} will be used to avoid clutter. 
To be precise, we should write | 


{wE2:&(w) Ee B} 
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in place of {€ € B}. Incidentally, {€ € B} is just a convenient way of writing 
the inverse image €~! (B) of a set. 


Definition 1.4 


The o-field o (€) generated by a random variable € : {2 — R consists of all sets 
of the form {€ € B}, where B is a Borel set in R. 


Definition 1.5 


The o-field o {€; : i € I} generated by a family {€; : i € I} of random variables 
is defined to be the smallest o-field containing all events of the form {& € B}, 
where B is a Borel set in R and € J. 


Exercise 1.3 


We call f : R > R a Borel function if the inverse image f~t (B) of any Borel 
set B in R is a Borel set. Show that if f is a Borel function and € is a random 
variable, then the composition f (€) is o (€)-measurable. 


Hint Consider the event {f (€) € B}, where B is an arbitrary Borel set. Can this 
event be written as {E € A} for some Borel set A? 


Lemma 1.2 (Doob—Dynkin) 


Let £ be a random variable. Then each ø (€)-measurable random variable 7 can 
be written as 


n= f(E) 


for some Borel function f : R > R. 


The proof of this highly non-trivial result will be omitted. 


Definition 1.6 
Every random variable £ : N —> R gives rise to a probability measure 
P(B) = P{£ € B} 


on R defined on the o-field of Borel sets B € B(R). We call Pe the distribution 
of €. The function Fẹ : R > [0, 1] defined by 


F; (x) = P{E < 2} 


~ 
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Exercise 1.4 


Show that the distribution function Fẹ is non-decreasing, right-continuous, and 


lim Fe (x) = 0, „im Fe (x) = 1. 


t — — OO 


Hint For example, to verify right-continuity show that F¢(zn) —> F(x) for any de- 
creasing sequence £n such that tn — x. You may find the results of Exercise 1.1 
useful. 


Definition 1.7 
If there is a Borel function fe : R — R such that for any Borel set B C R 
P{£ € B}= Jf (x) dz, 


then € is said to be a random variable with absolutely continuous distribution 
and fe is called the density of £. If there is a (finite or infinite) sequence of 


pairwise distinct real numbers 21,272,... such that for any Borel set B C R 
P{€EB}= y P{E=a5}, 
ziE€B 
then € is said to have discrete distribution with values z1,22,... and mass 


P{€ = x; } at Ti. 


Exercise 1.5 
Suppose that € has continuous distribution with density fe. Show that 


d 


aa re (=) = fe (2) 


if fe is continuous at zx. 
Hint Express Fe (x) as an integral of fe. 


Exercise 1.6 


Show that if € has discrete distribution with values 71, £2,... , then Fẹ is 
constant on each interval (s,¢] not containing any of the z;’s and has jumps of 
size P {E = 2;} at each z;. 


Hint The increment Fẹ (t) — Fẹ (s) is equal to the total mass of the z;’s that belong 
to the interval [s, t). 
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Jefinition 1.8 


The joint distribution of several random variables é&,...,én is a probability 
measure P¢,,...¢, on R” such that 


Pey...é, (B) = P{(&y--+s€n) € BY 


or any Borel set B in R”. If there is a Borel function fe., : R” -> R such 
hat 


or any Borel set B in R”, then fe,,.¢, is called the joint density of &1,..., En. 


n 


Definition 1.9 
\ random variable € : {2 > R is said to be integrable if 
J Ke dP < oo. 
R 
Chen 
BE) = | ¢4P 
R 


xists and is called the expectation of €. The family of integrable random vari- 
ibles € : N — R will be denoted by L! or, in case of possible ambiguity, by 
ERP: 


=-xample 1.3 


Che indicator function 14 ofa set A is equal to 1 on A and 0 on the complement 
2\ A of A. For any event A 


E (14) a ládP a= P(A). 
2 
Ne say that 7: R 4 R is a step function if 
= So nila: 
t=! 


vhere 71,...,%n are real numbers and Aj,..., Án are pairwise disjoint events. 


Chen : 7 
B(n)= | ndP= Son f 1a: dP = Yo mP (Ai). 
i=l i=l 
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Exercise 1.7 
Show that for any Borel function h : R — R such that h (€) is integrable 


E(h(€)) = [ h(x) dP; (2) 


Hint First verify the equality for step functions h : R — R, then for non-negative ones 
by approximating them by step functions, and finally for arbitrary Borel functions by 
splitting them into positive and negative parts. 


In particular, Exercise 1.7 implies that if € has an absolutely continuous 
distribution with density fe, then 


+00 


E(h(€)) = J h(x) fe (2) dz. 


— OO 


If € has a discrete distribution with (finitely or infinitely many) pairwise distinct 
values £1, £2,... , then 


E (h(€)) = dh (ai) P {£ = zi}. 


Definition 1.10 


A random variable € : N > R is called square integrable if 


I £|” dP < o0. 
2 


Then the variance of € can be defined by 
var(é) = | (€ - EE)? dP. 
Q 
The family of square integrable random variables € : {2 — R will be denoted 


by L?(N, F, P) or, if no ambiguity is possible, simply by L°. 


Remark 1.2 
The result in Exercise 1.8 below shows that we may write /(€) in the definition 


of variance. 


Exercise 1.8 


Show that if € is a square integrable random variable, then it is integrable. © 
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Hint Use the Schwarz inequality 
[E (En)]? < E (€*) E (n?) © (11) 


with an appropriately chosen 7. 


Exercise 1.9 


Show that if 7 : 2 — [0, 00) is a non-negative square integrable random vari- 
able, then 


Eln’) = 2 J tP(n > t) dt. 


Hint Express E(n’) in terms of the distribution function F,,(t) of n and then integrate 
by parts. 


1.3 Conditional Probability and Independence 


Definition 1.11 


For any events A, B € F such that P (B) 4 0 the conditional probability of A 


given B is defined by 
P(ANB) 


P (AIB) = Som 


Exercise 1.10 


Prove the total probability formula 
P(A) = P(A|Bi)P(B1) + P(A|B2)P(B2) +--> 


for any event A € F and any sequence of pairwise disjoint events B1, Bo,...€ F 
such that B, U Ba U---= Q and P(B,,) # 0 for any n. 


Hint A=(ANBi)U(ANB2)U-:-. 


Definition 1.12 
Two events A, B € F are called independent if 
P(AN B) = P(A)P(B). 
In general, we say that n events A,,..., An E F are independent if 


P(A; N Aiz eTA) = P(A; )P (Ai) ++: P(Ai, ) 
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for any indices 1 < tı <19 <- Cig IN. 


Exercise 1.11 


Let P(B) # 0. Show that A and B are independent events if and only if 
P(A|B) = P(A). 


Hint If P(B) #0, then you can divide by it. 


Definition 1.13 


Two random variables € and 7 are called independent if for any Borel sets 
A, B € B(R) the two events 


{EEA} and {ne B} 


are independent. We say that n random variables €,,...,&, are independent if 
for any Borel sets B,,..., Bn E B (R) the events 


{£ E By},...,{&, € Bn} 


are independent. In general, a (finite or infinite) family of random variables 
is said to be independent if any finite number of random variables from this 
family are independent. 


Proposition 1.1 


If two integrable random variables £, n : 2 — R are independent, then they are 
uncorrelated, i.e. 


E(&n) = E(&€)E(n), 


provided that the product éņ is also integrable. If &,...,€, : R + R are 
independent integrable random variables, then 


E (E162 -+ En) = E (61) E (€62) E (En), 


provided that the product )&-:-&, is also integrable. 


Definition 1.14 


Two o-fields G and H contained in F are called independent if any two events 


AEG and BEH 
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are independent. Similarly, any finite number of o-fields Gj,...,G, contained 
in F are independent if any n events 


Aj EG,,...,An E Gn 


are independent. In general, a (finite or infinite) family of o-fields is said to be 
independent if any finite number of them are independent. 


Exercise 1.12 


Show that two random variables € and 7 are independent if and only if the 
o-fields o (£) and a (yn) generated by them are independent. 


Hint The events in o (£) and ø (7) are of the form {€ € A}, and {7 € B}, where A 
and B are Borel sets. 


Sometimes it is convenient to talk of independence for a combination of 
random variables and o-fields. 


Definition 1.15 
We say that a random variable € is independent of a o-field G if the o-fields 
a(€) and G 


are independent. This can be extended to any (finite or infinite) family con- 
sisting of random variables or a-fields or a combination of them both. Namely, 
such a family is called independent if for any finite number of random variables 
€1,...,&m and o-fields G,,...,G, from this family the o-fields 


Oi iiss (Gn) iem Ga 


are independent. 


1.4 Solutions 


Solution 1.1 
If Ay C Ag C:--, then 
A; UAgU--: = Ay U (Aa \ 41) U (43 \ Ag) Ue: 
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where the sets A1, A2 \ A1, A3 \ Ao,... are pairwise disjoint. Therefore, by the 
definition of probability measure 
P(A, UAgU---) = P(A; U(A2 \ 41) U (Ap \ A2) U- ) 
P(A) +P (Ag \ Ai) +P (As \ Ao) +o 
lim P (An). 


| 


| 


The last equality holds because the partial sums in the series above are 
P (Ai) + P (A2 \ Ay) +--+ +P (An \ An-1) = P(4 U- -U An) 
aP Anh 
If Ay D Ap D--- , then the equality 
P(A, NAAN.) = lim P (An) 
follows by taking the complements of An and applying De Morgan’s law 
R\ (Ai N ALN ++) = (R \ Ar) U (2 \ Ad) U>. 

Solution 1.2 


Since B, is a contracting sequence of events, the results of Exercise 1.1 imply 
that 


P(ByN Ban --:) = lim P (Ban) 

lim P(A U Anat es -) 

N— OO 

lim (P (An) + P (Angi) ++) 
0. 


lA 


The last equality holds because the series }°>”-_, P(An) is convergent. The 
inequality above holds by the subadditivity property | 


P (An U Angi U) < P (An) + P (Anyi) +. 
It follows that P (BA B2N---)=0. 


Solution 1.3 


If B is a Borel set in R and f : R > R is a Borel function, then f~! (B) is also 
a Borel set. Therefore 


{f (€) € B} = {£ € f~ (B)} 
belongs to the o-field o (£) generated by £. It follows that the composition f (€) 


ic 7 (£)-measurahle 
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Solution 1.4 
If x < y, then {€ < £} C {E < y}, so 
Fg (£) = P {E < £} < P {E< y} = Fy (y). 


This means that F¢ is non-decreasing. 
Next, we take any sequence zı > z2 > --» and put 


lim Zp = T. 
n> 00 


Then the events 
CELTES nje 


form a contracting sequence with intersection 
{€ <z} = {E < 21} A {E < r2} N. 
It follows by Exercise 1.1 that 
Fe (xz) = P{E<zr}= lim P 16524) = lim Fe (Zn). 


This proves that F¢ is right-continuous. 
Since the events 


(ee ee fae eee 


form a contracting sequence with intersection @ and 


{E<1}c{E< 2c 
form an expanding sequence with union 2, it follows by Exercise 1.1 that 
lim F (2) = lim F (-n) = lim P{€ < -n} = P (0) =0, 
jim F; (x) = Jim F; (n) = lim P 16m np =P (2)=—1, 


since F¢ is non-decreasing. 


Solution 1.5 


If € has a density fg, then the distribution function Fẹ can be written as 


Fe(z)=P{E<s}= | fel) ay. 
Therefore, if fe is continuous at x, then Fẹ is differentiable at z and 


4 Fy (2) = fe (2). 
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Solution 1.6 
If s < t are real numbers such that x; ¢ (s,t] for any 2, then 
F(t) — Fe(s) = P{E < t} - P{E < s} = P{E € (s,t]} = 0, 


i.e. Fe(s) = Fe(t). Because Fẹ is non-decreasing, this means that Fẹ is constant 
on (s,¢]. To show that Fẹ has a jump of size P{€ = x;} at each z;, we compute 


Jim Fe(t) — lim F(s) = jim P{€ < t} - lim P{E < s} 
P{€ < zi} — P{E < ai} = P{E = a}. 


Solution 1.7 


If h is a step function, 
n 
h= S hila,, 
i=1 


where hi,...,hn are real numbers and 4ı,..., An are pairwise disjoint Borel 
sets covering R, then 


E (h(E = Mel La, (€)) = DMP {ee A) 


Fanus as ) dF; (x ) = f h(a) dP; (2). 


Next, any non-negative Borel function h can be approximated by a non- 
decreasing sequence of step functions. For such an h the result follows by the 
monotone convergence of integrals. Finally, this implies the desired equality 
for all Borel functions h, since each can be split into its positive and negative 
parts, h = ht — h`, where ht,h~ >Q. 


Solution 1.8 

By the Schwarz inequality (1.1) with ņ = 1, if € is square integrable, then 
[E (lg) = [E (11E)? < E (1°) E (£) = E () < œ, 

i.e. € is integrable. 


Solution 1.9 
Let F(t) = P {n < t} be the distribution function of 7. Then 


E(n*) = J ü t?dF(t). 
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Since P(ņ > t) = 1 — F(t), we need to show that 


f dF (t) =2 f t0- F) dt (1.2) 
0 0 


First, let us establish a version of (1.2) with oo replaced by a finite number a. 
Integrating by parts, we obtain 


: 2dF = Í 2d( F(t) — 
J: (t) [ PaF 1) 


0 


(F(t) -= 1)|, - 2 [ t(F(t) —1)dt 
= —a?(1- F(a)) +2 a t(1 — F(t))dt. (1.3) 


We see that (1.2) follows from (1.3), provided that 
a? (1—F(a)) 30, asa—> o. (1.4) 
But 
0 <a’ (1-— F(a)) =a°P(n > a) < (n+1)?P(n > n) < 4n?P(n >n), 
where n is the integer part of a, and 
Dns 2 
1 eee > N n dP < oo. 


Hence, 


Pin) s | nap => | n’ dP > 0 (1.5) 
{n>n} kan Y {k<n<k+1} 


as n — œ, which proves (1.4). 
Solution 1.10 
Since BUBU- =R, 
A= AN(B,UB,U---) =(ANB,)U(ANBg)U-:: , 


where 
(AN Bi) N (AN Bj) =AN(B;NB;) =AND=8. 


By countable additivity 
P(A) 


P((AN By) U(AN Ba) U:->) 
P(ANB,) + P(AN Bo) +- 


I! 


1. Review of Probability 15 


Solution 1.11 
If P(B) #0, then A and B are independent if and only if 


= P(ANB) 

PIA) = PB) 

In turn, this equality holds if and only if P(A) = P (A| B). 

Solution 1.12 

The ø-fields o (€) and ø (7) consist, respectively, of events of the form 


{EE A} and {ne B}, 


where A and B are Borel sets in R. Therefore, o (€) and o (7) are independent 
if and only if the events {€ € A}, and {n € B} are independent for any Borel 
sets A and B, which in turn is equivalent to € and 7 being independent. 


2 


Conditional Expectation 


Conditional expectation is a crucial tool in the study of stochastic processes. 
It is therefore important to develop the necessary intuition behind this notion, 
the definition of which may appear somewhat abstract at first. This chapter is 
designed to help the beginner by leading him or her step by step through several 
special cases, which become increasingly involved, culminating at the general 
definition of conditional expectation. Many varied examples and exercises are 
provided to aid the reader’s understanding. 


2.1 Conditioning on an Event 


The first and simplest case to consider is that of the conditional expectation 
E (|B) of a random variable € given an event B. 


Definition 2.1 


For any integrable random variable € and any event B € F such that P(B) #0 
the conditional expectation of € given B is defined by 


BUI) = Bom [ £ dP. 
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Example 2.1 


Three coins, 10p, 20p and 50p are tossed. The values of those coins that land 
heads up are added to work out the total amount £. What is the expected total 
amount £ given that two coins have landed heads up? 

Let B denote the event that two coins have landed heads up. We want to 
find E (€|B). Clearly, B consists of three elements, 


B = {HHT, HTH, THH}, 


each having the same probability ¿. (Here H stands for heads and T for tails.) 
The corresponding values of £ are 


¢(HHT) = 10 + 20 = 30, 


€(HTH) = 10+50 = 60, 
€(THH) = 20 + 50 = 70. 


Therefore 
1 t /30 60 70 
E(é\B) = —— dP = + | = + — + — ) = 53 
(IB) aim | Arrr) 3 


Exercise 2.1 


Show that E (E[N) = E(€). 


Hint The definition of E (£) involves an integral and so does the definition of E (€|2). 
How are these integrals related? 
Exercise 2.2 


Show that if 
1 frweA 


La(w) = { 0 forw GA 
(the indicator function of A), then 
E(14|B) = P(AIB), 
where 
P(AN B) 
P(B) 
is the conditional probability of A given B. 


P(A|B) = 


Hint Write fp 14 dP as P(AN B). 
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2.2 Conditioning on a Discrete Random 
Variable 


The next step towards the general definition of conditional expectation involves 
conditioning by a discrete random variable 7 with possible values y1, ya, ... such 
that P{n = yn} Æ 0 for each n. Finding out the value of 7 amounts to finding 
out which of the events {7 = yn} has occurred or not. Conditioning by 7 should 
therefore be the same as conditioning by the events {7 = yn}. Because we do 
not know in advance which of these events will occur, we need to consider all 
possibilities, involving a sequence of conditional expectations 


E(Elin=}), E lin =ye}),.... 


A convenient way of doing this is to construct a new discrete random variable 
constant and equal to E (E| {7 = yn}) on each of the sets {7 = yn}. This leads 
us to the next definition. 


Definition 2.2 


Let € be an integrable random variable and let 7 be a discrete random variable 
as above. Then the conditional expectation of € given n is defined to be a random 
variable E(€|n) such that 


E(E\n)(w) = E(El {n = yn}) if nw) = yn 


for any n = 1,2,.... 


Example 2.2 


Three coins, 10p, 20p and 50p are tossed as in Example 2.1. What is the 
conditional expectation E (€|n) of the total amount € shown by the three coins 
given the total amount 7 shown by the 10p and 20p coins only? | 

© Clearly, 7 is a discrete random variable with four possible values: 0, 10, 20 
and 30. We find the four corresponding conditional expectations in a similar 
way as in Example 2.1: | 


E (£| {n =O})=25, E(é|{n = 10}) = 35, 

E (£| {n = 20}) = 45, E(€|{n = 30}) = 55. 

Therefore 
| | 25 if n(w) = 0, 

J 35 if n(w) = 10, 

55 if n(w) = 30. 
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Example 2.3 


Take @ = [0,1] with the o-field of Borel sets and P the Lebesgue measure on 
[0, 1]. We shall find E(€|7) for 


2 ife (4,3 
0 ifze (4,1). 


Clearly, 7 is discrete with three possible values 1, 2,0. The corresponding events 
are 


{7 = 1} = (0, ae 
{n = 2} = (3,3), 
{7 = 0} = (2, 1] 


For x € (0, §] 


1 2 
BEME) = EEO S) = F | dade = 2 
For x € ($, =) 


14 


1 2 SE 1 $ 2 = 
EEMG) = EEG =F J, 2de = z 


And for x € (2,1] 


; Oif 38 
EEM) = EEGI = 7 J, arden = 2. 


The graph of E(€|n) is shown in Figure 2.1 together with those of € and 7. 


Exercise 2.3 
Show that if 7 is a constant function, then E(é|n) is constant and equal to 


EK). 


Hint The event {n = c} must be @ or N for any cE R. 


Exercise 2.4 


Show that P(A|B) if B 
I w €E 
E(1a|1B)(w) = l P(A|22 \ B) if w ¢ B 


for any B such that 1 # P(B) £0. 
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Figure 2.1. The graph of E (€|n) in Example 2.3 


Hint How many different values does 1g take? What are the sets on which these 
values are taken? 


Exercise 2.5 


Assuming that 7 is a discrete random variable, show that 


E(E(€\n)) = E(é). 


J, E(¢|n) dP = J dP 


for any event B on which ņ is constant. The desired equality can be obtained by 
covering N by countably many disjoint events of this kind. 


Hint Observe that 


Proposition 2.1 


If € is an integrable random variable and ņ is a discrete random variable, then 


1) E (Eln) is o (ņn)-measurable; 


2) For any A € ø (n) | | 
J E(é|n) dP = J édP (2.1) 
A A 


Proof 


Suppose that 7 has pairwise distinct values yi, y2,... . Then the events 


{n= y1}, {0 = y2},.-. 
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are pairwise disjoint and cover 92. The o-field o (7) is generated by these events, 
in fact every A € ø (7) is a countable union of sets of the form {7 = yn}. Because 
E (€|7) is constant on each of these sets, it must be ø (7)-measurable. | 

For each n we have 


J BaP = j E(El {N = yn}) dP 
{n=yn} {n=yn } 


= J EdP. 
{n=yn} 


Since each A € o(n) is a countable union of sets of the form {n = yn}, which 
are pairwise disjoint, it follows that 


| E&maP = J. é dP, 


as required. O 


2.3 Conditioning on an Arbitrary Random 
Variable 


>roperties 1) and 2) in Proposition 2.1 provide the key to the definition of the 
‘onditional expectation given an arbitrary random variable n. 


Definition 2.3 


et € be an integrable random variable and let 7 be an arbitrary random 
ariable. Then the conditional expectation of € given 7 is defined to be a random 
ariable E(€|n) such that 


1) E (Eln) is o (n)-measurable; 
2) For any A € a(n) 


J eemar = f aP. 


'emark 2.1 


ľe can also define the conditional probability of an event A € F given 7 by 


P (Aln) = E(1aln), 


here 14 is the indicator function of A. 


2. Conditional Expectation 23 


Do the conditions of Definition 2.3 characterize E (€\n) uniquely? The 
lemma below implies that E(€|n) is defined to within equality on a set of 
full measure. Namely, 


if E€ = €' a.s., then E (Eln) = E (E'|n) a.s. (2.2) 


The existence of E (€|7) will be discussed later in this chapter. 


Lemma 2.1 


Let (2, F, P) be a probability space and let G be a o-field contained in F. If € 
is a G-measurable random variable and for any B € G 


f gap =o, 
B 


then € = 0 a.s. 


Proof 


Observe that P {€ >«} =0 for any € > 0 because 
O<eP{E>e}= eaP < | EdP =0. 
{E>} {€>¢} 


The last equality holds, since {€ > e} € G. Similarly, P {€ < —e} = 0 for any 
€ > 0. As a consequence, | 


P{-e<xE<e}=1 


for any € > 0. 
Let us put 
An = {-4 <€< 3}. 


Then P (An) = 1 and 
{€ =0} = [) An. 


n=l 


Because the An form a contracting sequence of events, it follows that 
P{ċ¢=0}= lim P(An)=1, 
n= 


as required. O 
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One difficulty involved in Definition 2.3 is that no explicit formula for E (€|7) 
is given. If such a formula is known, then it is usually fairly easy to verify 
conditions 1) and 2). But how do you find it in the first place? The examples 
and exercises below are designed to show how to tackle this problem in concrete 


cases. 


Example 2.4 


Take 2 = [0,1] with the o-field of Borel sets and P the Lebesgue measure on 
[0, 1]. We shall find E(é|n) for 


2 if x € [0, >), 
T if x € [5,1]. 


ele) = 207, nle) = 4 


Here ņ is no longer discrete and the general Definition 2.3 should be used. 
First we need to describe the o-field a(n). For any Borel set B C [5,1] we 
have 
B= in € B} € a(n) 


and 
(0, 5) UB = {n € B} U {n = 2} € o (1). 
In fact sets of these two kinds exhaust all elements of o (7). The inverse image 
{n € C} of any Borel set C C R is of the first kind if 2 ¢ C and of the second 
kind if 2 € C. 
If E(E|n) is to be o (ņ)-measurable, it must be constant on [0, $) because 7 
is. If for any z € [0, 5) 


E(€|n)(x) = E(é|[0, §)) 
1 
= P({0,4)) hp” 


1l 


1 f2 
= F 2r? dr 
0 


~a 


Dl wir 


then 


[Bema =] EE) de 


0,3) (0,5) 


i.e. condition 2) of Definition 2.3 will be satisfied for A = (0, 5). 
Moreover, if E(é|7) = € on [5,1], then of course 


| EEmME)de = fea) dx 
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for any Borel set B C [ż, 1). 
Therefore, we have found that 


1 if xz 0, ) 
E(€\|n)(x) = l Sa? if x i H J 


Because every element of o(7) is of the form B or [0, 1) UB, where B C [5,1] 
is a Borel set, it follows immediately that conditions 1) and 2) of Definition 2.3 
are satisfied. The graph of E(€|n) is presented in Figure 2.2 along with those 
of € and p. 





Figure 2.2. The graph of E (€\7) in Example 2.4 


Exercise 2.6 


Let 2 = [0,1] with Lebesgue measure as in Example 2.4. Find the conditional 
expectation E (€|n) if 


E(x) = 22°, n(x) = 1 — [22 — 1|. 


Hint First describe the o-field generated by n. Observe that 7 is symmetric about 5. 
What does it tell you about the sets in ø (n)? What does it tell you about E (€|n) if it 
is to be o (ņ)-measurable? Does it need to be symmetric as well? For any A in a (n) 
try to transform f ,&64P to make the integrand symmetric. 


Exercise 2.7 


Let 2 be the unit square [0,1] x [0,1] with the o-field of Borel sets and P the 
Lebesgue measure on [0, 1] x [0,1]. Suppose that € and 7 are random variables 
on £2 with joint density 

Teale, y) =X+Y 
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for any x,y € [0,1], and fe n(£, y) = 0 otherwise. Show that 


— 2437 
— 346 





E (En) 


Hint It suffices (why?) to show that for any Borel set B 


J cap = | ao dP 
{n€B} Gea, To 


Try to express each side of this equality as an integral over the square [0,1] x [0,1] 
using the joint density fe,,(z, y). 





Exercise 2.8 


Let 92 be the unit square (0, 1} x [0, 1] with Lebesgue measure as in Exercise 2.7. 
Find E (€|n) if € and 7 are random variables on {2 with joint density 


3 
féen(a,y) = 5 (z? + y”) 
for any x,y € [0,1], and fé n(z, y) = 0 otherwise. 


Hint This is slightly harder than Exercise 2.7 because here we have to derive a formula 
for the conditional expectation. Study the solution to Exercise 2.7 to find a way of 
obtaining such a formula. 


Exercise 2.9 


Let N be the unit disc {(a, y) : £? +y? < 1} with the o-field of Borel sets and 
P the Lebesgue measure on the disc normalized so that P (Q) = 1, i.e. 


P (A) = |j dedy 


for any Borel set A C f2. Suppose that € and ņ are the projections onto the z 
and y axes, 
E(z,y) =z,  n(z,y)=y 


for any (x,y) € 92. Find E (€|n). 


Hint What is the joint density of € and 7? Use this density to transform the integral 


f €? dP 
{nEB} 


for an arbitrary Borel set B so that the integrand becomes a function of 7. How is 
this function of ņ related to E (£ In) ? 
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2.4 Conditioning on a o-Field 


We are now in a position to make the final step towards the general definition 
of conditional expectation. It is based on the observation that E (|) depends 
only on the o-field o (7) generated by 7, rather than on the actual values of 77. 


Proposition 2.2 
If a(n) = a(n’), then E(€|n) = E(€|n') a.s. (Compare this with (2.2).) 


Proof 
This is an immediate consequence of Lemma 2.1. O 
Because of Proposition 2.2 it is reasonable to talk of conditional expectation 


given a o-field. The definition below differs from Definition 2.3 only by using an 
arbitrary o-field G in place of a o-field o (7) generated by a random variable n. 


Definition 2.4 


Let € be an integrable random variable on a probability space (2,7, P), and 
let G be a o-field contained in F. Then the conditional expectation of € given 
G is defined to be a random variable E (€|G) such that 


1) E(€|G) is G-measurable; 
2) For any AEG 


J E(€\G) dP = J EdP. (2.3) 
A A | 

Remark 2.2 

The conditional probability of an event A € F given a o-field G can be defined 
by 


P(A|G) = E (14|6) 


where 1, is the indicator function of A. 


The notion of conditional expectation with respect to a o-field extends 
conditioning on a random variable 7 in the sense that | 


E (f\o(n)) = E (€n), 


where o(7) is the o-field generated by n. 
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Proposition 2.3 


E(€|G) exists and is unique in the sense that if € = €' a.s., then E(E|G) = 
E(E'|G) a.s. 


Proof 


Existence and uniqueness follow, respectively, from Theorem 2.1 below and 
Lemma 2.1. O 


Theorem 2.1 (Radon—Nikodym) 


Let (2,7, P) be a probability space and let G be a o-field contained in F. 
Then for any random variable € there exists a G-measurable random variable 


C such that 
J, EdP = Í CaP 


The Radon—Nikodym theorem is important from a theoretical point of view. 
However, in practice there are usually other ways of establishing the existence 
of conditional expectation, for example, by finding an explicit formula, as in 
the examples and exercises in the previous section. The proof of the Radon- 
Nikodym theorem is beyond the scope of this course and is omitted. 


for each A EG. 


Exercise 2.10 
Show that if G = {@, 2}, then E(€|G) = E(€) as. 


Hint What random variables are G-measurable if G = {@, 2}? 
Exercise 2.1] 
Show that if € is G-measurable, then E(€|G) = £ as. 


Hint The conditions of Definition 2.4 are trivially satisfied by € if € is G-measurable. 


Exercise 2.12 
Show that if B € G, then 
E (E (|G) |B) = E (€|B). 


Hint The conditional expectation on either side of the equality involves an integral 
aver R. How are these integrals related to one another? 
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2.5 General Properties 


Proposition 2.4 
Conditional expectation has the following properties: 

1) E(ag + 0¢|G) = aE (|G) + bE(¢|G) (linearity); 

2) E(E(E|G)) = E(g); 

3) E(EC|G) = €E(C\G) if € is G-measurable (taking out what is known); 


4) E(€|G) = E(€) if € is independent of G (an independent condition drops 
out); | 


5) E(E(€|G)|H) = E(E|H) if H C G (tower property); 
6) If E > 0, then E(E|G) > 0 (positivity). 


Here a,b are arbitrary real numbers, €,¢ are integrable random variables on a 
probability space (N, F, P) and G, H are o-fields on (2 contained in F. In 3) we 
also assume that the product €¢ is integrable. All equalities and the inequalities 
in 6) hold P-a.s. 


Proof 
1) For any BEG 


J (aE (EG) + bE(C|G)) dP = a J E(é|G) dP + b J E(¢|G) dP 
B B B 


af eap+b/ cap 
J (og + 0c) ap 


By uniqueness this proves the desired equality. 

2) This follows by putting A = N in (2.3). Also, 2) is a special case of 5) 
when H = {@, Q}. | 

3) We first verify the result for € = 14, where A € G. In this case 


Í 14E(n|G)dP = J E(n|G) dP 
B ANB 


= J n dP 
ANB 


i 1andP 
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for any B € G, which implies that 
L4E(n|G) = ECanl|9) 


by uniqueness. In a similar way we obtain the result if € is a G-measurable step 


m 
E = ) Glas; 
j=l 


where A; € G for j = 1,...,m. Finally, the result in the general case follows 
by approximating € by G-measurable step functions. 

4) Since € is independent of G, the random variables € and 1p are inde- 
pendent for any B e G. It follows by Proposition 1.1 (independent random 
variables are uncorrelated) that 


J 2® dP 


function, 


E(€)E(1B) 
E(€1g) 


J, EdP, 


i 


| 


which proves the assertion. 
5) By Definition 2.4 


J EEO aP = J EdP 


[ E(é\H) dP = f € dP 


for every B € H. Because H C G it follows that 


J E(&JG) dP = / E(EJH) dP 
B B 


for every B € G, and 


for every B € H. Applying Definition 2.4 once again, we obtain 
E(E(€|G)|H) = E(E|H). 


6) For any n we put 


An =f EEO <->}. 


Then A, € G. If £ > 0 a.s., then 


o</ £dP= | EQGIg) dP < -ŽP (An), 
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which means that P (An) = 0. Because 
{E (EIG) < 0} = LJ An 
n=l 


it follows that 
P {E (|G) < 0} = 0, 


completing the proof. Q 

The next theorem, which will be stated without proof, involves the notion 
of a convex function, such as max (1,x) or e~*, for example. In this course the 
theorem will be used mainly for |z|, which is also a convex function. In general, 
we call a function y: R > R conves if for any z,y € R and any à € (0, 1] 


pAr + (1—-A)y) < Av (z) + (1—- A) ely). 


This condition means that the graph of ọ lies below the cord connecting the 
points with coordinates (z,y(z)) and (y, y (y)). 


Theorem 2.2 (Jensen’s Inequality) 


Let y : R > R be a convex function and let € be an integrable random variable 
on a probability space (N, F, P) such that py (£) is also integrable. Then 


p (E (E1G)) < E(y(g)|9) as. 


for any o-field G on N contained in F. 


2.6 Various Exercises on Conditional 
Expectation 


Exercise 2.13 


Mrs. Jones has made a steak and kidney pie for her two sons. Eating more 
than a half of it will give indigestion to anyone. While she is away having tea 
with a neighbour, the older son helps himself to a piece of the pie. Then the 
younger son comes and has a piece of what is left by his brother. We assume 
that the size of each of the two pieces eaten by Mrs. Jones’ sons is random and 
uniformly distributed over what is currently available. What is the expected 
size of the remaining piece given that neither son gets indigestion? 
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Hint All possible outcomes can be represented by pairs of numbers, the portions of 
the pie consumed by the two sons. Therefore N can be chosen as a subset of the 
plane. Observe that the older son is restricted only by the size of the pie, while the 
younger one is restricted by what is left by his brother. This will determine the shape 
of N. Next introduce a probability measure on (2 consistent with the conditions of 
the exercise. This can be done by means of a suitable density over §2. Now you are in 
a position to compute the probability that neither son will get indigestion. What is 
_ the corresponding subset of §2? Finally, define a random variable on §2 representing 
the portion of the pie left by the sons and compute the conditional expectation. 


Exercise 2.14 


As a probability space take 2 = [0,1) with the o-field of Borel sets and the 
Lebesgue measure on [0,1). Find E (€|n) if 


2r fr0<zr<żŁ 
= 2r? = 2 A 


Hint What do events in ø (7) look like? What do o (n)-measurable random variables 
look like? If you devise a neat way of describing these, it will make the task of 
finding E (€|7) much easier. You will need to transform the integrals in condition 2) 
of Definition 2.3 to find a formula for the conditional expectation. 


Exercise 2.15 


Take 2 = [0,1] with the o-field of Borel sets and P the Lebesgue measure on 
(0, 1]. Let 
n(x) = z(1— z) 


for x € [0,1]. Show that 


E(€|n)(x) = Se 


for any x € [0,1]. 
Hint Observe that 7 (x) = 7 (1 — x). What does it tell you about the o-field generated 


by n? Is 5 (€ (x) + € (1 — x)) measurable with respect to this o-field? If so, it remains 
to verify condition 2) of Definition 2.3. 


Exercise 2.16 
Let €,7 be integrable random variables with joint density f¢,»(z, y). Show that 


= i= T fenkt, n) dz 


ASINS fr fen (2,0) dx 


Hint Study the solutions to Exercises 2.7 and 2.8. 
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Remark 2.3 
If we put 
_ Fen 2; y) 
fe sn (zly) aE fr (y) , 
where 


AE [ fem(a,y) de 


is the density of n, then by the result in Exercise 2.16 


E (Gin) = | 2 fen (ein) ae 
We call fe, (z|y) the conditional density of € given n. 
Exercise 2.17 
Consider L? (F) = L? (N, F, P) as a Hilbert space with scalar product 
L? (F) x L? (F) > (€, C) > EEC) E R. 


Show that if € is a random variable in L? (F) and G is a ø-field contained in 
F, then E(E|G) is the orthogonal projection of € onto the subspace L’ (G) in 
L? (F) consisting of G-measurable random variables. 


Hint Observe that condition 2) of Definition 2.4 means that € — E(E|G) is orthogonal 
(in the sense of the scalar product above) to the indicator function 14 for any A € G. 


2.7 Solutions 


Solution 2.1 | 
Since P (N) = 1 and fp EdP = E(E), 


E (|N) = pg | §aP = BO. 


Solution 2.2 
By Definition 2.1 


E(14|B) 


lI 
S 
by 
y 

> 

Y 
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P(AN B) 
P(B) 
P(A|B). 


| 


Solution 2.3 


Since 7 is constant, it has only one value c € R, for which 
in=c} = 22. 
Therefore E (€|n) is constant and equal to 
E(\n)(w) = E (€| in = c}) = B(E|2) = E (8) 
for each w € N. The last equality has been verified in Exercise 2.1. 


Solution 2.4 


The indicator function 1g takes two values 1 and 0. The sets on which these 
values are taken are 


{ig=1}=B, {lp=0}=N\B. 


Thus, for w € B 
E(14|ls)(w) = E(14|B) = P(A|B), 


as in Exercise 2.2. Similarly, for w € N \ B 
E(lalla)(w) = E|? \ B) = P(A|2 \ B). 
Solution 2.5 


First we observe that 


[ Beier = f (sm eaP) dP = | ¢aP (2.4) 


for any event B. 

Since 7 is discrete, it has countably many values y),y2,.... We can as- 
sume that these values are pairwise distinct, i.e. y; A yj if i Æ j. The sets 
{n = y1}, {1 = ye2},... are then pairwise disjoint and they cover the whole 
space 2. Therefore, by (2.4) 


E(E(é|n)) = J E(Eln) dP 
= E = Ynt) dP 
3 La (él{ = yn}) 
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| 

& 
~ 
AN 
5r 


Solution 2.6 


First we need to describe the o-field o (7) generated by 7. Observe that ņ is 
symmetric about 5, 

n(x) = (1-2) 
for any x € [0,1]. We claim that ø (7) consists of all Borel sets A C [0,1] 
symmetric about $, i.e. such that 


isis (2.5) 
Indeed, if A is such a set, then A = {n € B}, where 
B = {2x : x € AN [0, 4]} 


is a Borel set, so A € a(n). On the other hand, if A € ø (n), then there is a 
Borel set B in R such that A = {7 € B}. Then 


LEA & n(x)EB 
& n(l-r)eEeB 
+ ļ]-rE€A, 


so A satisfies (2.5). 
We are ready to find E (€|n). If it is to be ø (n)-measurable, it must be 
symmetric about Ł, i.e. 


E (Ejn) (x) = E (Eln) (1 — 2) 


for any z € [0,1]. For any A € ø (ny) we shall transform the integral below so 
as to make the integrand symmetric about 3: 


f 20? aa = [ears | Pax 
A A A 
= fèar) (1—2)* dz 
A 1—A 
[tart f a-a) dz 
A A 


f G + (1- z)”) dx. 
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Figure 2.3. The graph of E (€|7) in Exercise 2.6 


It follows that 
E (ln) (z) = 2? + (1-2), 
The graphs of £, 7 and E (€|n) are shown in Figure 2.3. 


Solution 2.7 


Since 
{n € B} = [0,1] x B 


for any Borel set B, we have 


fremt” 7 [ fe tenen) dx dy 
j J: (s Ia ts) A 
- [ € + zy) dy 


2+ 3N | - | [3 y 
dz d 
joa 3 + 6y Fen(@,y) dx dy 
2+ 3y J 
= r+y)dz |d 
EAR ) ) 
1 | 
= f, (+39) @ 


Because each event in a (7) is of the form {n € B} for some Borel set B, this 
means that condition 2) of Definition 2.3 is satisfied. The random variable are 


and 
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is a (7)-measurable, so condition 1) holds too. It follows that 


2+ 37 
E (ln) = 36, 





Solution 2.8 


We are looking for a Borel function F : R —> R such that for any Borel set B 


J ¿dP = J F (n) dP. (2.6) 
{nEB} {nEB} 


Then E (Eln) = F (n) by Definition 2.3. 

We shall transform both integrals above using the joint density fé n (z, y) 
in much the same way as in the solution to Exercise 2.7, except that here we 
do not know the exact form of F (y). Namely, 


[cap = ff 2 henley) aedy 
{nEB} BJR 
=5/ (/ e (2 +1?) ae] dy 
2JB \ Jion 
3 SE 
= ah (G t3 ) dy 


J JE y) fen(x,y) dx dy 
= 3j F off T yf) de) ay 
= 5J PO) (5 +07] dy. 


Then, (2.6) will hold for any Borel set B if 


and 


J F (n) dP 
{nEB} 


stay 3+ 6y? 





F = = ——__—_, 

W) ety? 4+12? 
It follows that a 
+ 07) 

E(&\n) = F (n) = 44 12n? 


Solution 2.9 


We are looking for a Borel function F : R —> R such that for any Borel set 


BCR 
Í €? dP = / F (n) dP. (2.7) 
{nEB} {nEB} 
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Then, by Definition 2.3 we shall have E (£*|n) = F (n). 

Let us transform both sides of (2.7). To do so we observe that the ran- 
dom variables € and 7 have uniform joint distribution over the unit disc 
oe {(z,y) r? + y? < 1}, with density 

l 
fésn (x,y) = = 


if x? +y° <1, and fe n (z, y) = 0 otherwise. It follows that 


J €* dP J ia fen (x,y) dx dy 
{nE B} BJR 
ig eee 
= =j x” dx dy 
TIBI-VS1\—y? 
2 


daly De _ ,2\3/2 


and 


hem? = [ [rw fen (@,y) dz dy 


-1f ro [Odea 
= - | F (y) Q- dy. 
B 


If (2.7) is to be satisfied for all Borel sets B, then 


F(y) =; (1-9); 


This means that 
E (€*|\n) (2,y) = F (n (z,y)) = F (y) = 


for any (x,y) in R. 
Solution 2.10 


If G = {@, 2}, then any constant random variable is G-measurable. Since 


[ cap=BO©= | EG dP 


[cap =0= | E® dP, 


i+ fallanroe that FEIC — AL) ae ae raniired 


aur) 


wi = 


and 


2. Conditional Expectation 39 


Solution 2.11 


Because the trivial identity 


| cap = f cap 


holds for any A € G and € is G-measurable, it follows that E(€|G) = £ a.s. 


Solution 2.12 
By Definition 2.3 
| EES) ap = f aP, 
B B 


since B € G. It follows that 


E (E (EIG) |B) = PTE [ E(é|G) dP 


= E (|B). 


Solution 2.13 


The whole pie will be represented by the interval [0,1]. Let x € [0,1] be the 
portion consumed by the older son. Then [0,1 — zx] will be available to the 
younger one, who takes a portion of size y € [0,1 — x]. The set of all possible 
outcomes is 

N = {(x,y):z,y > 0,5 +y <1}. 


The event that neither of Mrs. Jones’ sons will get indigestion is 


1 
A={(eyeQiau< 5h, 


These sets are shown in Figure 2.4. If x is uniformly distributed over [0,1] and 
y is uniformly distributed over [0,1 — z], then the probability measure P over 


N with density 


will describe the joint distribution of outcomes (z, y), see Figure 2.5. 
Now we are in a position to compute 


P(A) = J f(x,y) de dy 


1 1 
fr "edud 
o Jo 1-2 
in /2 
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Figure 2.3. The graph of E (€|7) in Exercise 2.6 


It follows that 
E (Eln) (x) = z? + (1-2). 


The graphs of £, 7 and E (€|n) are shown in Figure 2.3. 
Solution 2.7 


Since 
{n € B} = [0,1] x B 


for any Borel set B, we have 


Jaen me h: Je fen(z,y) dz dy 


LAL 


He +) de) dy 
1 1 
= 5 +50) dy 
Lo? 


2+ 3n Fi 
dP = aoe y)dzd 
Lark B Rot Oy Fen(@ y) ai 


2+ 3y / 
= x+y) dx | dy 
EAI ) ) 
1 1l 
= | (=4+ =y) dy. 
i, (ata) Y 


Because each event in o (7) is of the form {n € B} for some Borel set B, this 
means that. condition 2) of Definition 2.3 is satisfied. The random variable $+32 


and 








2. Conditional Expectation 37 


is ø (7)-measurable, so condition 1) holds too. It follows that 


2+ 37 
E (En) = 5 ey 


Solution 2.8 


We are looking for a Borel function F : R —> R such that for any Borel set B 


J cdr = J F (n} dP. (2.6) 
{nEB} {nEB} 


Then E (Eln) = F (n) by Definition 2.3. 

We shall transform both integrals above using the joint density fe n (£, y) 
in much the same way as in the solution to Exercise 2.7, except that here we 
do not know the exact form of F (y). Namely, 


[cap = [ef a tenleydray 
{nEB} BYR 
=5/ (/ +(e +) ae dy 
2JB \ Jion 
3 i ie 
~ sS Gta ) ay 


Jan O oe J fe y) fen (x,y) dx dy 
7 Ii TA a +y”) ts) a 
E I w (5+ y? ) dy 


Then, (2. 6 will hold for any Borel set B if 


and 


it ay" B _ 3+ by" 





F = —_—_—_ , 
It follows that ae 
+ 07) 
E = F = oa, 
(Eln) e + [are 


Solution 2.9 


We are looking for a Borel function F : R —> R such that for any Borel set 


BCR 
J € dP = J F (n) dP. (2.7) 
{nEB} {nE B} 
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Then, by Definition 2.3 we shall have E (€7|n) = F (n). 

Let us transform both sides of (2.7). To do so we observe that the ran- 
dom variables € and 7 have uniform joint distribution over the unit disc 
2 = { (x,y) (2? y < 1}, with density 

Fen (x,y) are 


if x° +y? <1, and fe n (x,y) = 0 otherwise. It follows that 


Í M nak a dx dy 
nE 
= hex x” dz dy 
J/1—y? =y? 


= _ 9\3/2 


and 


I? = [ [rw fen (t,y) dz dy 


= rof dz dy 
z = J Fw) (1-4)? dy. 


If (2.7) is to be satisfied for all Borel sets B, then 
1 


~(1—y7). 


F(y)=3 


This means that 
E (Ein) (2,y) = F (n(2,y)) = F (y) = 


for any (x,y) in 9. 


(1—y*) 


eo] = 


Solution 2.10 


If G = {@, 2}, then any constant random variable is G-measurable. Since 


| cap =B@)= | EG aP 


[cap =0= [ae dP, 


it follows that E(EIG) = Ef) a.s.. as reauired. 


and 
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Solution 2.11 


Because the trivial identity 
/ é dP = J EdP 
A A 
holds for any A € G and € is G-measurable, it follows that E(E|G) = £ a.s. 


Solution 2.12 
By Definition 2.3 


J EEO dP = | cap, 


since B € G. It follows that 


E (E (EG) |B) PIES [ E (|G) dP 


1 
j i Kai 
E(E|B). 


Solution 2.13 


The whole pie will be represented by the interval [0,1]. Let x € [0,1] be the 
portion consumed by the older son. Then [0,1 — x] will be available to the 
younger one, who takes a portion of size y € [0,1 — z]. The set of all possible 
outcomes is 

N = {(z,y):z,y > 0,£ +y <1}. 


The event that neither of Mrs. Jones’ sons will get indigestion is 
| 1 
A= lene ay < H 


These sets are shown in Figure 2.4. If z is uniformly distributed over [0, 1] and 
y is uniformly distributed over [0,1 — zx}, then the probability measure P over 
N with density 





f(z,y) = — 


will describe the joint distribution of outcomes (x,y), see Figure 2.5. 
Now we are in a position to compute 


P(A) = | f(w,y) dz dy 


> 72 1 
=| f dz dy 
0 0 l-r 
2 


In V2. 
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Figure 2.4. The sets N and A in Exer- 
cise 2.13 


Figure 2.5. The density f (x,y) in Exer- 
cise 2.13 





The random variable 


E(z,y) =l-a2-y 
defined on N represents the size of the portion left by Mrs. Jones’ sons. Finally, 
we find that 
1 


BIA) = rp J, (1—2—y) f(,y) dx dy 


1 2 21—-zr-y 
—z | J l-z 4 


1— ln v2 
In4 ` 





Solution 2.14 


The ø-field o (7) generated by 7 consists of sets of the form B U (B + 5) for 
some Borel set B C (0, 5). Thus, we are looking for a ø (7)-measurable random 
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variable Ç such that for each Borel set B C (0, 5) 


_ (dir= Eau C (x) dz. (2.8) 


Then E (Eln) = Ç by Definition 2.3. 
Transforming the integral on the left-hand side, we obtain 


J E(x) dz = J 2x7 dz +j 2z’ dz 
BU(B+4) B B+} 


| estar f 2043) dx 
B B 
2 | (2? + (x + 4)’) dz. 

B 


For Ç to be ø (7)-measurable it must satisfy 


((z) =C(z+35) (2.9) 


for each x € [0, 4). Then 


Jaspi ® dP = [cw et j dx 
= | codes | cer) dz 
[o d+ f ce ao 


2 J eae 
B 
If (2.8) is to hold for any Borel set B C [0, 4), then 


C(x) =a? + (2 + 4)’ 


for each x € [0, 4). The values of ¢ (x) for x € [5, 1) can be obtained from (2.9). 
It follows that 

x? + (x +4)° for0<2z< $, 

(a — 4)" +2? for$ <r<1. 


E (&\n) (z) = ¢ (2) sd 


The graphs of €, 7 and E (E|ņ) are shown in Figure 2.6. 
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E(€|n) 





Figure 2.6. The graph of E (€|7) in Exercise 2.14 


Solution 2.15 


Since n(x) = n(1 — z), the o-field a(n) consists of Borel sets B C [0,1] such 


that 
B=1-B, 


where 1 — B = {1 — x : x € B}. For any such B 


[ e)a = 5 | elde+ 5 f (a)az 
= zJ, eede E(1 — 2x) dz 
= 5 | ee) )dz + = heee 


7 Í E(x) nan 


Because 3 (€ (x) + £ (1 — x)) is ø (n)-measurable, it follows that 


E (€m) (z) = SO HSU 2) 


Solution 2.16 
We are looking for a Borel function F (y) such that 


J aP = | F (n) dP 
{neB} {n€B) 


for any Borel set B in R. Because F (n) is o (n)-measurable and each event 
in ø (ņ) can be written as {7 € B} for some Borel set B, this will mean that 


merl Mor 
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Let us transform the two integrals above using the joint density of € and 7: 


Jan f? ~ AET (x,y) dz dy 
~ f (ftne) dz) d 


fren” al =f fr y) fen (t,y) dx dy 
= [Fw (ften (x,y) dz) dy. 


If these two integrals are to be equal for each Borel set B, then 


Jetson (x,y) dx 
Ja fen (x,y) dr 


and 


F (y) = 


It follows that 
fo T fen (z,n) dx 


E (Eln) = F (n) = fo fe (zn) dr ` 
R JEn (T, 


Solution 2.17 


We denote by ¢ the orthogonal projection of € onto the subspace L? (G) C 
L? (F) consisting of G-measurable random variables. Thus, € — ¢ is orthogonal 
to L? (G), that is, 
El(é - 07] = 
for each y € L’ (G). But for any A € G the indicator function 14 belongs to 
L? (G), so 
EE — ¢)1a] = 0. 
Therefore 


| ear = E(€1,4) = E(¢1a) = [ cap 
for any A € G. This means that ¢ = E(E€|G). 


a 


Martingales in Discrete Time 


3.1 Sequences of Random Variables 


A sequence €;,&2,... of random variables is typically used as a mathematical 
model of the outcomes of a series of random phenomena, such as coin tosses 
or the value of the FTSE All-Share Index at the London Stock Exchange on 
consecutive business days. The random variables in such a sequence are indexed 
by whole numbers, which are customarily referred to as discrete time. It is 
important to understand that these whole numbers are not necessarily related 
to the physical time when the events modelled by the sequence actually occur. 
Discrete time is used to keep track of the order of events, which may or may 
not be evenly spaced in physical time. For example, the share index is recorded 
only on business days, but not on Saturdays, Sundays or any other holidays. 
Rather than tossing a coin repeatedly, we may as well toss 100 coins at a time 
and count the outcomes. 


Definition 3.1 


The sequence of numbers €; (w) , £2 (w),... for any fixed w € N is called a 
sample path. 


A sample path for a sequence of coin tosses is presented in Figure 3.1 (+1 
stands for heads and —1 for tails). Figure 3.2 shows the sample path of the 
FTSE All-Share Index up to 1997. Strictly speaking the pictures should con- 
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-1 


Figure 3.1. Sample path for a sequence of coin tosses 








Figure 3.2. Sample path representing the FTSE All-Share Index up to 1997 


sist of dots, representing the values ĉ (w) , éz (w),... , but it is customary to 
connect them by a broken line for illustration purposes. 


3.2 Filtrations 

As the time n increases, so does our knowledge about what has happened in 
the past. This can be modelled by a filtration as defined below. 

Definition 3.2 

A sequence of o-fields F1, F2,... on R such that 


Fi C FoC::-CF 
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is called a filtration. 


Here Fn represents our knowledge at time n. It contains all events A such 
that at time n it is possible to decide whether A has occurred or not. As n 
increases, there will be more such events A, i.e. the family Fn representing our 
knowledge will become larger. (The longer you live the wiser you become!) 


Example 3.1 


For a sequence £é, £2,... of coin tosses we take F,, to be the o-field generated 


by Ciena ws 
Fn =o (&1,---,€n)- 


Let 
A = {the first 5 tosses produce at least 2 heads}. 


At discrete time n = 5, i.e. once the coin has been tossed five times, it will be 
possible to decide whether A has occurred or not. This means that A € Fs. 
However, at n = 4 it is not always possible to tell if A has occurred or not. If 
the outcomes of the first four tosses are, say, 


tails, tails, heads, tails, 


then the event A remains undecided. We will have to toss the coin once more 
to see what happens. Therefore A ¢ F4. 

This example illustrates another relevant issue. Suppose that the outcomes 
of the first four coin tosses are 


tails, heads, tails, heads. 


In this case it is possible to tell that A has occurred already at n = 4, whatever 
the outcome of the fifth toss will be. It does not mean, however, that A belongs 
to F4. The point is that for A to belong to F4 it must be possible to tell whether 
A has occurred or not after the first four tosses, no matter what the first four 
outcomes are. This is clearly not so in the example in hand. 


Exercise 3.1 


Let £1,&2,... be a sequence of coin tosses and let Fn be the o-field generated 
by £1,.--,€n. For each of the following events find the smallest n such that the 
event belongs to Fn: 

A = {the first occurrence of heads is preceeded by no more than 10 tails}, 

B = {there is at least 1 head in the sequence &, &2,...}, 

C = {the first 100 tosses produce the same outcome}, 


ND — Sthoroa ara nn mnra than 9 hoade and 9 taila amnnoa tha firat K tnecacl 
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Hint To find the smallest element in a set of numbers you need to make sure that 
the set is non-empty in the first place. 


Suppose that €, &,... is a sequence of random variables and F1, F2,... is 
a filtration. A priori, they may have nothing in common. However, in practice 
the filtration will usually contain the knowledge accumulated by observing the 
outcomes of the random sequence, as in Example 3.1. The condition in the def- 
inition below means that 7, contains everything that can be learned from the 


values of €;,...,&€,- In general, it may and often does contain more information. 
Definition 3.3 
We say that a sequence of random variables €), €2,...is adapted to a filtration 


Fi,F.2,... if En is F,-measurable for each n = 1,2,.... 


Example 3.2 


If Fan = 0(&,...,&n) is the o-field generated by &,...,&,, then 1, &,... is 
adapted to Fi, Fo,.... 


Exercise 3.2 


Show that 
Fn te eee oy 


is the smallest filtration such that the sequence €,,&,... is adapted to Fı, 
Fy,.... That is to say, if G|,G2,... is another filtration such that €,, £,... is 
adapted to G,,Go,... , then Fn C Gn for each n. 


Hint For o(1,...,&n) to be contained in Gn you need to show that &1,...,&n are 
Gn-measurable. 


3.3 Martingales 


The concept of a martingale has its origin in gambling, namely, it describes 
a fair game of chance, which will be discussed in detail in the next section. 
Similarly, the notions of submartingale and supermartingale defined below are 
related to favourable and unfavourable games of chance. Some aspects of gam- 
bling are inherent in the mathematics of finance, in particular, the theory of 
financial derivatives such as options. Not surprisingly, martingales play a cru- 
cial role there. In fact, martingales reach well beyond game theory and appear 
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in various areas of modern probability and stochastic analysis, notably, in dif- 
fusion theory. First of all, let us introduce the basic definitions and properties 
in the case of discrete time. 


Definition 3.4 


A sequence €, €2,... of random variables is called a martingale with respect to 
a filtration F1, Fo,... if 


1) £n is integrable for each n = 1,2,... ; 
2) £,,&9,... is adapted to Fi, Fo,...; 
3) E(En41|Fn) = En a.s. for each n = 1,2,.... 


Example 3.3 


Let 71,72,... be a sequence of independent integrable random variables such 
that E(nn) = 0 for all n = 1,2,.... We put 


En = m+: +n; 
Fn = o(m,---5%n). 


Then £n» is adapted to the filtration Fn, and it is integrable because 


E(|Enl) = Em +--+ + ml) 
< E(|m|) +--+ + Eml) 
< ©. 


Moreover, 


E(EntilFn) = E(tntilFn) + E(EnlFn) 
= E(mn+i) + En 
= on 
since 741 is independent of Fn (‘and independent condition drops out’) and 


En is F,-measurable (‘taking out what is known’). This means that €, is a 
martingale with respect to Fn. 


Example 3.4 


Let € be an integrable random variable and let F1, F2,... be a filtration. We 


put 
En = E(E\Fn) 
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forn = L2 
Then én is Fn-measurable, 


which implies that 


E(|En|) < E(E(lE] |Fn)) = EUSI) < 0, 


and 
B(Enii|Fn) = E(E(E|Fn4i)|Fn) == E(E|Fn) = Cas 


since Fa C Fn41 (the tower property of conditional expectation). Therefore En 
is a martingale with respect to Fn. 


Exercise 3.3 


Show that if €, is a martingale with respect to Fn, then 
E(fi) = E) =+ 
Hint What is the expectation of E(fn41|Fn)? 


Exercise 3.4 


Suppose that €, is a martingale with respect to a filtration Fn. Show that En 
is a martingale with respect to the filtration 


Gn = o(&,. es cee 
Hint Observe that Gn C Fn and use the tower property of conditional expectation. 
Exercise 3.5 
Let €, be a symmetric random walk, that is, 


En =m tN, 


where 71,72,... is a sequence of independent identically distributed random 
variables such that i 
Pim = 1} = Pim = -1}=3 


(a sequence of coin tosses, for example). Show that €2 — n is a martingale with 
respect to the filtration 
Fn = a(m,- ars Mn). 
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Hint You want to transform E (E244 —(n+1) \F n) to obtain £2 — n. Write 


2 2 
Eny = (En + 1n41) 

2 2 
Mn4+1 + 2Nn+1 En F En 
and observe that En is Fn-measurable, while mn+1 is independent of Fn. To transform 
the conditional expectation you can ‘take out what is known’ and use the fact that 


‘an independent condition drops out’. Do not forget to verify that £2 — n is integrable 
and adapted to Fn. 


Exercise 3.6 


Let En be a symmetric random walk and Fn the filtration defined in Exer- 
cise 3.5. Show that 
Cn = (—1)" cos(mn) 


is a martingale with respect to Fn. 


Hint You want to transform E((—1)"*' cos(w&n41)|Fn) to obtain (—1)" cos(7€n). 
Use a similar argument as in Exercise 3.5 to achieve this. But, first of all, make sure 
that Çn is integrable and adapted to Fn. 


Definition 3.5 


We say that &),&,... is a supermartingale (submartingale) with respect to a 
filtration F1, Fo,... if 


1) En is integrable for each n = 1,2,... ; 
2) £,,€2,... is adapted to Fi, Fo,...; 
3) E(En+ilFn) < En (respectively, B(€n41|Fn) > En) a.s. for each n = 1,2,.... 


Exercise 3.7 


Let En be a sequence of square integrable random variables. Show that if €, is 
a martingale with respect to a filtration Fn, then £? is a submartingale with 
respect to the same filtration. 


Hint Use Jensen’s inequality with convex function y (x) = 2”. 


3.4 Games of Chance 


Suppose that you take part in a game such as the roulette, for example. Let 
m.m.... be a sequence of integrable random variables. where n. are vour 
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winnings (or losses) per unit stake in game n. If your stake in each game is 
one, then your total winnings after n games will be 


En =m e+. 


We take the filtration 
Fama isra 


and also put éo = 0 and Fo = {0, 2} for notational simplicity. 
If n — 1 rounds of the game have been played so far, your accumulated 
knowledge will be represented by the o-field Fn-1. The game is fair if 


E(En|Fn-1) = €n-1, 


that is, you expect that your fortune at step n will on average be the same as 
at step n — 1. The game will be favourable to you if 


E(En|Fn-1) > Cais 
and unfavourable to you if 
PGi Feat) < En-1 


for n = 1,2,.... This corresponds to €, being, respectively, a martingale, a 
submartingale, or a supermartingale with respect to Fn, see Definitions 3.4 
and 3.5. 

Suppose that you can vary the stake to be a, in game n. (In particular, an 
may be zero if you refrain from playing the nth game; it may even be negative if 
you own the casino and can accept other people’s bets.) When the time comes 
to decide your stake an, you will know the outcomes of the first n — 1 games. 
-= Therefore it is reasonable to assume that a, is 7,_1-measurable, where Fn—1 
represents your knowledge accumulated up to and including game n — 1. In 
particular, since nothing is known before the first game, we take Fo = {0, Q}. 


Definition 3.6 


A gambling strategy a,,Q2,... (with respect to a filtration F1, F2,...) is a 
sequence of random variables such that a, is #,_,-measurable for each n = 
1,2,..., where Fo = {0, 2}. (Outside the context of gambling such a sequence 
of random variables a, is called previsible.) 


If you follow a strategy a,,a2,..., then your total winnings after n games 
will be 


Cri = QAM +: + AnNa 
= Qı (È — fa) +e +n (En — En-1)- 
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We also put Co = O for convenience. 

The following proposition has important consequences for gamblers. It 
means that a fair game will always turn into a fair one, no matter which gam- 
bling strategy is used. If one is not in a position to wager negative sums of 
money (e.g. to run a casino), it will be impossible to turn an unfavourable 
game into a favourable one or vice versa. You cannot beat the system! The 
boundedness of the sequence a, means that your available capital is bounded 
and so is your credit limit. 


Proposition 3.1 
Let a1,Q2,... be a gambling strategy. 


1) If a1, @2,... is a bounded sequence and £o, £1, &2,... is a martingale, then 
Co, ¢1,¢2,... is a martingale (a fair game turns into a fair one no matter 
what you do); 


2) Ifa ,,Qa2,...is anon-negative bounded sequence and £o, €1, €2,... 1S a super- 
martingale, then Co, ¢1, ¢2,... is a supermartingale (an unfavourable game 
turns into an unfavourable one). 


3) If a,,Q@2,... is a non-negative bounded sequence and £o, 1, 2,... is a sub- 
martingale, then o, (1, C2,... is a submartingale (a favourable game turns 
into a favourable one). | 


Proof 


Because a, and C,- 1 are F,-1-measurable, we can take them out of the ex- 
pectation conditioned on F,_, (‘taking out what is known’, Proposition 2.4). 
Thus, we obtain 


E (GnlFn-1) = E (Cn-1 + On (En — En-1) |Fn-1) 
Cia + An (E (EnlFn-1) B Cac) . 


If €, is a martingale, then 
An (E (En|Fn-1) — En-1) = 9, 

which proves assertion 1). If £, is a supermartingale and a, > 0, then 
An (E (En|Fn-1) — En-1) < 0, 

proving assertion 2). Finally, assertion 3) follows because 
An (E (En|Fn-1) — En-1) > 0 


if £,, is a submartingale and a, > 0. O 
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3.5 Stopping Times 


In roulette and many other games of chance one usually has the option to quit 
at any time. The number of rounds played before quitting the game will be 
denoted by r. It can be fixed, say, to be r = 10 if one decides in advance 
to stop playing after 10 rounds, no matter what happens. But in general the 
decision whether to quit or not will be made after each round depending on 
the knowledge accumulated so far. Therefore 7 is assumed to be a random 
variable with values in the set {1,2,...}U {oo}. Infinity is included to cover the 
theoretical possibility (and a dream scenario of some casinos) that the game 
never stops. At each step n one should be able to decide whether to stop playing 
or not, i.e. whether or not 7 = n. Therefore the event that 7 = n should be 
in the o-field Fn representing our knowledge at time n. This gives rise to the 
following definition. 


Definition 3.7 


A random variable 7 with values in the set {1,2,...}U {oo} is called a stopping 
time (with respect to a filtration Fn) if for each n = 1,2,... 


{r =n} € Fy. 


Exercise 3.8 

Show that the following conditions are equivalent: 
1) {r < n} € Fn for each n = 1,2,...; 

2) {T =n} E€ Fn for each n =1,2,.... 


Hint Can you express {r < n} in terms of the events {r = k}, where k = 1,... n? 
Can you express {r = n} in terms of the events {r < k}, where k = 1,...,n? 


Example 3.5 (First hitting time) 


Suppose that a coin is tossed repeatedly and you win or lose £1, depending 
on which way it lands. Suppose that you start the game with, say, £5 in your 
pocket and decide to play until you have £10 or you lose everything. If ĉn is 
the amount you have at step n, then the time when you stop the game is 


T =min{n: én = 10 or 0}, 


and is called the first hitting time (of 10 or 0 by the random sequence £n). It 


SCE MR tg cee eee ee + tana tho filtratinn 
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Fn =o(E1,..-,&n). This is because 
{r=n}={0<& <10}N---N{0 < éEn-1 < 10} N {ên = 10 or 0}. 


Now each of the sets on the right-hand side belongs to Fn, so their intersection 
does too. This proves that 


for each n, so T is a stopping time. 


Exercise 3.9 


Let En be a sequence of random variables adapted to a filtration Fn and let 
B C R bea Borel set. Show that the time of first entry of En into B, 


T=min{n:€, € B} 
is a stopping time. 


Hint Example 3.5 covers the case when B = (—oo, 0] U[10, 00). Extend the argument 
to an arbitrary Borel set B. 


Let én be a sequence of random variables adapted to a filtration Fn and 
let r be a stopping time (with respect to the same filtration). Suppose that 
En represents your winnings (or losses) after n rounds of a game. If you decide 
to quit after 7 rounds, then your total winnings will be €,. In this case your 
winnings after n rounds will in fact be ran. Here a A b denotes the smaller of 


two numbers a and 8, 
a \ b= min (a,b). 


Definition 3.8 


We call Ean the sequence stopped at r. It is often denoted by £7. Thus, for 
each w € {2 
E(w) Pe Er(w)an (w). 


Exercise 3.10 


Show that if é„ is a sequence of random variables adapted to a filtration Fn, 
then so is the sequence éran. 


Hint For any Borel set B express {ran € B} in terms of the events {ĉx € B} and 
{T = k}, where k =1,...,n. 
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We already know that it is impossible to turn a fair game into an unfair 
one, an unfavourable game into a favourable one, or vice versa using a gambling 
strategy. The next proposition shows that this cannot be achieved using a 
stopping time either (essentially, because stopping is also a gambling strategy). 


Proposition 3.2 
Let 7 be a stopping time. 


1) If En is a martingale, then so is éran. 
2) If €, is a supermartingale, then so is Eran. 


3) If ¿n is a submartingale, then so is éran. 


Proof 


This is in fact a consequence of Proposition 3.1. Given a stopping time 7, we 


put 
1 ifr >n, 
Aan = : 
0 ifr<n. 


We claim that a, is a gambling strategy (that is, an is #,~,-measurable). This 
is because the inverse image {an € B} of any Borel set B C R is equal to 


O E€ Fanı 


if 0,1 ¢ B, or to 
N E Fanı 


if 0,1 € B, or to 

Grlj =r 2n >n] E 
if 1 € B and 0 ¢ B, or to 

{an =0} = {Tr <n} = {Tr <n- 1} € Fn- 
if 1 ¢ B and 0 € B. For this gambling strategy 

Eran = Q1 (1 — E0) ++: + an (En — En-1) . 


Therefore Proposition 3.1 implies assertions 1), 2) and 3) above. O 
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Example 3.6 


(You could try to beat the system if you had unlimited capital and unlimited 
time.) The following gambling strategy is called ‘the martingale’. (Do not con- 
fuse this with the general definition of a martingale earlier in this section.) 
Suppose a coin is flipped repeatedly. Let us denote the outcomes by 71, 72,... , 
which can take values +1 (heads) or —1 (tails). You wager £1 on heads. If you 
win, you quit. If you lose, you double the stake and play again. If you win this 
time round, you quit. Otherwise you double the stake once more, and so on. 
Thus, your gambling strategy is 


e Ai ifn, = = Nn-1 = tails, 
” ) 0 otherwise. 


Let us put 
| Cn = M +2 +- +2" ny 


and consider the stopping time 
T = min {n : n = heads}. 


Then Cran will be your winnings after n rounds. It is a martingale (check it!). 

It can be shown that P {r < co} = 1 (heads will eventualiy appear in the 
sequence 11, N2,... with probability one). Therefore it makes sense to consider 
Cr. This would be your total winnings if you were able to continue to play the 
game no matter how long it takes for the first heads to appear. It would require 
unlimited time and capital. If you could afford these, you would be bound to 
win eventually because ¢, = 1 identically, since 


~] —-2-—. -277142 =1 


for any n. . 


Exercise 3.11 


Show that if a gambler plays ‘the martingale’, his expected loss just before the 
ultimate win is infinite, that is, 


E (Çr-1) = —00. 


Hint What is the probability that the game will terminate at step n, i.e. that T = n? 
If T = n, what is ¢,-1 equal to? This will give you all possible values of ¢-_; and 
their probabilities. Now compute the expectation of Çr-—1. 
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3.6 Optional Stopping ‘Theorem 


If En is a martingale, then, in particular, 


E(En) = E(&1) 


for each n. Example 3.6 shows that E(€,) is not necessarily equal to E(€,) for 
a stopping time T. However, if the equality 


E(&,) = E(£) 


does hold, it can be very useful. The Optional Stopping Theorem provides 
sufficient conditions for this to happen. 


Theorem 3.1 (Optional Stopping Theorem) 


Let En be a martingale and 7 a stopping time with respect to a filtration Fn 
such that the following conditions hold: 


l) T< œ a.s., 
2) €, is integrable, 
3) E(Enlir>n}) > 0 as n > ov. 


Then 
E(&,) = E(&i). 


Proof 


Because 


Er = Eran + (E = En) lir>n}» 
it follows that 
E(£-) g E(Eran) + E(E-14r>n}) =. E(Enlir>n}). 


Since éran is a martingale by Proposition 3.2, the first term on the right-hand 
side is equal to 


E(Eran) = E(&). 


The last term tends to zero by assumption 3). The middle term 


E(Elgesny) = X E(&lraey) 


k=n+1 
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tends to zero as n — oo because the series 
OO 
E(€,) = X E(Eltr=e}) 
k=1 


is convergent by 2). It follows that E(€,) = E(&,), as required. O 


Example 3.7 (Expectation of the first hitting time for a random walk) 


Let En be a symmetric random walk as in Exercise 3.5 and let K be a positive 
integer. We define the first hitting time (of +K by €,) to be 


T= min {n : |€,| = K}. 


By Exercise 3.9 7 is a stopping time. By Exercise 3.5 we know that €2 -n is a 
martingale. If the Optional Stopping Theorem can be applied, then 


E (È -r)=E(-1)=0. 
This allows us to find the expectation 
E(r) = E(&) = K’, 


since |€,| = K. 

Let us verify conditions 1)—3) of the Optional Stopping Theorem. 

1) We shall show that P{r = œ} = 0. To this end we shall estimate 
P{r > 2Kn}. We can think of 2Kn tosses of a coin as n sequences of 2K 
tosses. A necessary condition for T > 2Kn is that no one of these n sequences 
contains heads only. Therefore 

1 n 
P{r>2Kn}< (1- xx] —+ 0 
as n — oo. Because {7 > 2Kn} for n = 1,2,... is a contracting sequence of 
sets (i.e. {7 > 2Kn} D {7 > 2K (n+ 1)}), it follows that 


P4700} <= P(N {r > 2K) 


lim P{r>2Kn}=0, 


completing the argument. 
2) We need to show that 


E (E2 — r|) < œ. 
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Indeed, 


ET) = X o nP {r =n} 
"eo 2K 
= SO >) (2Khn +k) P{r =2Kn + k} 
on 
SOX 2K (n+ 1) P{r > 2Kn} 


n=0 k=1 


< 4K? 50 (n+ 1) G-r) 


n=0 
< Ow, 


IA 


since the series $`? (n + 1) q” is convergent for any q € (—1, 1). Here we have 
recycled the estimate for P {r > 2Kn} used in 2). Moreover, €? = K*, so 


PUCET Sb) eh 
K? + E(r) 


< Ow. 


3) Since €2 < K? on {T >n}, 
E (&.lgrsn}) < K°P {r > n} 40 
as n — oo. Moreover, 
E (nlir>n}) < E (Tlir>n}) > 0 


as n — oo. Convergence to 0 holds because E (7T) < oo by 2) and {7 > n} is 
a contracting sequence of sets with intersection {7 = co} of measure zero. It 
follows that 

E ((& -= n) lr>n}) 7 0, 


as required. 


Exercise 3.12 


Let €, be a symmetric random walk and F,, the filtration defined in Exer- 
cise 3.5. Denote by 7 the smallest n such that |n| = K as in Example 3.7. 
Verify that 

Cn = (—1)" cos [r (En + K)] 
is a martingale (see Exercise 3.6). Then show that ¢, and 7 satisfy the con- 
ditions of the Optional Stopping Theorem and apply the theorem to find 


or’ aTa 
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Hint The equality Ç+ = (—1)" is a key to computing E'[(—1)"] with the aid of the 
Optimal Stopping Theorem. The first two conditions of this theorem are either obvious 
in the case in hand or have been verified elsewhere in this chapter. To make sure that 
condition 3) holds it may be helpful to show that 


IE(Cn1{r>n})| < P{r > n}. 


Use Jensen’s inequality with convex function ọ (x) = |z| to estimate the left-hand 
side. Do not forget to verify that Çn is a martingale in the first place. 


3.7 Solutions 


Solution 3.1 


A belongs to Fıı, but not to Fio. The smallest n is 11. 

B does not belong to Fn for any n. There is no smallest n such that B € Fn. 
C belongs to Fioo, but not to Fo9. The smallest n is 100. 

Since D = §, it belongs to Fn for each n = 1,2,.... Here the smallest n is 1. 


Solution 3.2 


Because the sequence of random variables €), €2,... is adapted to the filtration 
Gi,G2,... , it follows that En is G,-measurable for each n. But 

Gi CC, 
SO €1,..-,€n are G,-measurable for each n. As a consequence, 


Fam OG eign) C Un 
for each n. 
Solution 3.3 
Taking the expectation on both sides of the equality 
En = E(En+ilFn), 


we obtain 


E(En) = E(E(En4ilFn)) = ElEn+1) | 
for each n. This proves the claim. 


Solution 3.4 


The random variables €, are integrable because €, is a martingale with respect 
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to Fn. Since Gn is the o-field generated by &,,...,&,, it follows that n is 
adapted to G,. Finally, since Gn C Fn, 


En = E(En|Gn) 
E(E(En+1|Fn)|Gn) 
E(En+11Gn) 


| 


by the tower property of conditional expectation (Proposition 2.4). This proves 
that En is a martingale with respect to Gp. 


Solution 3.5 


Because 
E -n= (mt +n) -n 


is a function of m,..., m, it is measurable with respect to the o-field Fn gen- 
erated by m,...,%n, i.e. E — n is adapted to Fn. Since 
fal = [m + Ml S Iml n] = n, 


it follows that 
E(\€ — nl) < EE) +n <n?’ +n < oœ, 


so €* — n is integrable for each n. Because 
ei = ae + 27n+16n + ae 


= where €, and £2 are F,-measurable and nn+ı is independent of Fn, we can use 
Proposition 2.4 (‘taking out what is known’ and ‘independent condition drops 
out’) to obtain 


ECG rtin) ae iA areas a a 2E(nn+1£n|Fn) + E(€.|Fn) 


E(n-41) + 26, E (n+) + a 
= 1+2é. 


This implies that 
EE 4 —n-Il|F,) = En =n, 


so €2 — n is a martingale. 


Solution 3.6 


Being a function of én, the random variable Çn is Fn-measurable for each n, 


3. Martingales in Discrete Time 63 


since n is. Because |¢,| < 1, it is clear that Çn is integrable. Because 7,41 is 
independent of F, and En is #,-measurable, it follows that 


E(Cn+11Fn) E ((-1)"** cos[r (En + M41] |Fn) 

= (-1)"*'E (cos (nên) cos (7741) |Fn) 
—(-1)"**E (sin (nên) sin (741) |Fn) 
(—1)"*! cos (mên) E (cos (1717n+1)) 
—(—1)"*? sin (nên) E (sin (117n41)) 
= (—1)” cos (7€,) 
= Cn; 


| 


using the formula 
cos (a + 8) = cosacos $ — sin asin £. 


To compute E (cos(m7n41)) and E (sin (7n41)) observe that nn41 = 1 or —1 
and 


cost = cos(—7) = —1, 
sina = sin(—7) = 0. 


It follows that Cn is a martingale with respect to the filtration Fn. 


Solution 3.7 
If €, is adapted to Fn, then so is €%. Since En = E (€n41|Fn) for each n and 
p (x) = x” is a convex function, we can apply Jensen’s inequality (Theorem 2.2) 
to obtain 

En = [E (EntilFn))” < E (Elfa) 
for each n. This means that £2 is a submartingale with respect to Fn. 
Solution 3.8 
1)=2). If r has property 1), then 

{r< n} € Fn 


and 
{r <n-1} E Fn-1 C Fn, 


SO 
{T=n}={r<n}\fr<n-1}E Fy. 


2)=>1). If r has property 2), then 
Ir =k} E€ F.C Fan 
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for each k = 1,...,n. Therefore 


enre e UUT Sn eT x. 


Solution 3.9 


If 
r= min {n : & € B}, 


then for any n 
{r =n} = {6&1 € B} NAN {En-1 € BEN {& € B}. 


Because B is a Borel set, each of the sets on the right-hand side belongs to the 
o-field Fn = o (&,...,€n), and their intersection does too. This proves that 
{T =n} € Fn for each n, so T is a stopping time. 


Solution 3.10 
Let B C R be a Borel set. We can write 


Ern EBk {n EBT > nyu U {E&E B,T=k}, 
k=1 


where 
{En E B, T >n} = {En E BJA{T >n} E Fn 


and for each k = 1,...,n 
1 €B,r=k}={& € B}N{r=k}€ Fy C Fy. 
It follows that for each n 
eee € B} € F i 


as required. 


Solution 3.11 


The probability that ‘the martingale’ terminates at step n is 
1 
P{r=n}=— 
{r=n}= = 
(n — 1 tails followed by heads at step n). Therefore 


OO 


E(¢,-1) = X Gn-1P {r =n} 


1 


3 
| 


See = 2723) L 


Qn 
A2] 
= -5 p o 


I 
Me 


= 
| 
— 
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Solution 3.12 


The proof that ¢, is a martingale is almost the same as in Exercise 3.6. We 
need to verify that Çn and 7 satisfy conditions 1)—3) of the Optional Stopping 
Theorem. 

Condition 1) has in fact been verified in Example 3.7. 

Condition 2) holds because |¢,| < 1, so E'(|¢;|) < 1 < oo. 

To verify condition 3) observe that |¢,,| < 1 for all n, so 


IE(Caltrsny)| < E(|Cn|lir>n}) 


is E(lir>n}) 
P{r >n}. 


/\ 


The family of events {7 > n}, n = 1,2,... is a contracting one with intersection 
{r = oo}. It follows that 


IE(Galtrony)| < Pye > n} N P{r ag oo} 
as n > oo. But 


Pit = œ} =0 


by 1), completing the proof. 
The Optional Stopping Theorem implies that 


E(&)=E(&) 
Because £, = K or —K, we have 
Cr = (1) cos[r(K + €,)] = (-1)’. 
Let us compute 
E(G) = —5 (cos [r (1 + K)] + cos [r (-1 + K))) 
= cos (rK) = (-1)". 


It follows that 
E[(-1)] = (-1)*. 
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Martingale Inequalities and Convergence 


Results on the convergence of martingales provide an insight into their structure 
and have a multitude of applications. They also provide an important inter- 
pretation of martingales. Namely, it turns out that a large class of martingales 
can be represented in the form 


En = E (€|Fn) ) (4.1) 


where € = limn €, is an integrable random variable and F1, F2,... is the filtra- 
tion generated by £1, &2,... , see Theorem 4.4 below. This makes it possible to 
think of &,,&,... as the results of a series of imperfect observations of some 
random quantity €. As n increases, the accumulated knowledge Fn about £ 
increases and £, becomes a better approximation, approaching the observed 
quantity € in the limit. 

We shall begin with a few classical inequalities for martingales, known as 
the Doob inequalities. They provide the tools we shall need to study the con- 
vergence of martingales and, later on, the properties of stochastic integrals. 
Then we shall present a classical result known as Doob’s Martingale Conver- 
gence Theorem, which provides the limit lim, n of a martingale. However, 
Doob’s theorem has one inconvenient feature. It guarantees only that én con- 
verges a.s., even though the limit is known to be an integrable random variable. 
However, to obtain (4.1) we need convergence in L}, which gives rise to a condi- 
tion called uniform integrability. This condition and its consequences, including 
(4.3), will be studied in the second section. Finally, as an example of an appli- 
cation reaching beyond the theory of martingales, we present an elegant proof 
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4.1 Doob’s Martingale Inequalities 


Proposition 4.1 (Doob’s maximal inequality) 


Suppose that €,,n € N, is a non-negative submartingale (with respect to a 
filtration Fn). Then for any A > 0 


AP (max > A) <E C TTT ' 


where 1, is the characteristic function of a set A. 


Proof 
We put é} = max Ek for brevity. For A > 0 let us define 
T=min{k <n: ék >A}, 


if there is a k < n such that E€ > A, and T = n otherwise. Then 7 is a stopping 
time such that 7 < n a.s. Since én is a submartingale, 


E(En) 2 E(€,). 


But 
E(&) = E (&\ex>ay) + E (Er les <a} ) - 


Observe that if €* > A, then é > A. Moreover, if &* < A, then T = n, and so 
€, = En. Therefore 


E(én) > EEr) > AP (E >A) +E (Enlges ay); 
It follows that 
AP (ce 2 x) < E (En) “E (Enles <y}) =E (Enlye~>y}) ) 


completing the proof. O 


Theorem 4.1 (Doob's maximal L? inequality) 
If Eén, n € N, is a non-negative square integrable submartingale (with respect 
to a filtration Fn), then 


| 2 
E|max€,| < 4E\én|’. (4.2) 





a 
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Proof 


Put €* = maxg<n Ek. By Exercise 1.9, Proposition 4.1, the Fubini theorem and 
finally the Cauchy—Schwarz inequality 


Bie 2 | tP (E > t) dt < 2 | E (Enler >t}) dt 


of ( J. $ a) t=? f 6 ( f” a) iP 


2 | éntndP = 2E (Ens) < 2 (E él) (Ble) 


| 


gna? 
Dividing by (£ E>] ) , we get (4.2). O 


The proof of Doob’s Convergence Theorem in the next section hinges on an 
inequality involving the number of upcrossings. 


Definition 4.1 


Given an adapted sequence of random variables ;, €2,... and two real numbers 
a < b, we define a gambling strategy ai, @2,... by putting 


Q1 = 0 
and for n = 1,2,... 
1 if a, = 0 and n <a, 
ai = 1 if a, = 1 and n < b, 
0 otherwise. 


It will be called the upcrossings strategy. Each k = 1,2,... such that a, = 1 
and &k+1 = 0 will be called an upcrossing of the interval [a,b]. The upcrossings 
form a (finite or infinite) increasing sequence 


Uj LUL « 


The number of upcrossings made up to time n, that is, the largest k such that 
uk < n will be denoted by Un[a, b] (we put Un[a, b] = 0 if no such k exists). 


The meaning of the above definition is this. Initially, we refrain from playing 
the game and wait until €, becomes less than a. As soon as this happens, we 
start playing unit stakes at each round of the game and continue until €, 
becomes greater than b. At this stage we refrain from playing again, wait until 
En becomes less than a, and so on. The strategy a, is defined in such a way 
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that a, = 0 whenever we refrain from playing the nth game, and an = 1 
otherwise. During each run of consecutive games with a, = 1 the process én 
crosses the interval [a,b], starting below a and finishing above b. This is what 
is meant by an upcrossing. Observe that each upcrossing will increase our total 
winnings by at least b — a. For convenience, we identify each upcrossing with 
its last step k, such that ag = 1 and ax; = 0. A typical sample path of the 
upcrossings strategy is shown in Figure 4.1. 


Exercise 4.1 


Verify that the upcrossings strategy a, is indeed a gambling strategy. 


Hint You want to prove that an is F,~-1-measurable for each n. Since the upcrossings 
strategy is defined by induction, a proof by induction on n may be your best bet. 


Lemma 4.1 (Upcrossings Inequality) 
If €;,€2,... is a supermartingale and a < b, then 
(b — a) E(U,,[a, b]) < E((& — a) ). 


By z7 we denote the negative part of a real number z, i.e. r~ = max {0, —z}. 


Proof 


Let | 
Cn = 1 (E1 — o) +--+ + On (En — En-1) 


be the total winnings at step n = 1,2,... if the upcrossings strategy is followed, 

see Figure 4.1. It will be convenient to put G9 = 0. By Proposition 3.1 (one 

cannot beat the system using a gambling strategy) Cn is a supermartingale. 
Let us fix an n and put k = U,[a, b], so that 


O < uy <U <L: LUkIN 
Clearly, each upcrossing increases the total winnings by b — a, 
Gui — Gui 20-4 
for i = 1,...,k. (We put uo = 0 for simplicity.) Moreover, 
Cn — Guz. 2 — (En-a) 


It follows that 
Cn > (b- a)U,,[a, b] — (En ~ a) . 
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Figure 4.1. Typical paths of En, an and Cn; upcrossings are indicated by bold lines 


taking the expectation on both sides, we get 
E (Çn) > (b — a) E(Unla, b]) — E((fn ~a) ). 
But Çn is a supermartingale, so 
0 = F(a) 2 E (Cn), 


which proves the Upcrossings Inequality. O 


4.2 Doob’s Martingale Convergence Theorem 


Theorem 4.2 (Doob's Martingale Convergence Theorem) 


Suppose that 1,&,... is a supermartingale (with respect to a filtration 
Fi, Fa, ...) such that 
sup E (\En|) < CO. 
n 


Then there is an integrable random variable £ such that 


lim £, =E as. 
n—> 00 
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Remark 4.1 


In particular, the theorem is valid for martingales because every martingale is a 
supermartingale. It is also valid for submartingales, since €, is a submartingale 
if and only if —€, is a supermartingale. 


Remark 4.2 


Observe that even though all the €, as well as the limit € are integrable random 
variables, it is claimed only that €, trends to € a.s. Note that no convergence 
in L! is asserted. 


Proof (of Doob’s Martingale Convergence Theorem) 


By the Upcrossings Inequality 


E ((€)—a)- 
E (U,[a, b]) < al a < oe < œ, 


where 
M = sup E (|&,|) < co. 


Since U,,[a, b] is a non-decreasing sequence, it follows that 


; — M + |a| 
OR) ea O er moar 
This implies that 
P f lim Unla, b] < o0 } =i 
for any a < b. Since the set of all pairs of rational numbers a < b is countable, 
the event 


A= A { lim Unla, b] < o0 } (4.3) 


a<b rational 


has probability 1. (The intersection of countably many events has probability 1 
if each of these events has probability 1.) 
We claim that the sequence €, converges a.s. to a limit €. Consider the set 


B = {liminf én < limsup&,} C 2 


on which the sequence &, fails to converge. Then for any w € B there are 
rational numbers a,b such that 


lim inf n(w) <a < b < limsupé,(w), 
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implying that limp... Un[a, b](w) = oo. This means that B and the event A in 
(4.3) are disjoint, so P(B) = 0, since P(A) = 1, which proves the claim. 

It remains to show that the limit € is an integrable random variable. By 
Fatou’s lemma 


E (lim inf Enl) 
lim inf E (En) 
sup Æ (lén|) < o0 


E (|8|) 


IA 


A 


This completes the proof. O 


| Exercise 4.2 


| Show that if €, is a non-negative supermartingale, then it converges a.s. to an 
\ integrable random variable. 


int To app 7 Doob’s Theorem all you need to verify is that the sequence €, 1s 
bounded in L’, i.e. the supremum of E (|&,|) is less than oo. 


S 


4.3 Uniform Integrability and L! Ceara 
of Martingales 


The conditions of Doob’s theorem imply pointwise (a.s.) convergence of martin- 

gales. In this section we shall study convergence in L!. To this end we introduce 

a stronger condition called uniform integrability. Proposition 4.2 shows that it 

is a necessary condition for L! convergence. In Theorem 4.2 we prove that uni- 

form integrability is in fact sufficient for a martingale to converge in Lt. This 

enables us to show that each integrable martingale is of the form E (E|Fn). As 
an application we give a martingale proof of Kolmogorov’s 0-1 law. 


Exercise 4.3 


Show that a random variable € is integrable if and only if for every € > 0 there 
exists an M > 0 such that | 


Í ll dP < €. 
{|€|>M} 


Hint Split Q into two sets: {|€ | > M} and {|é| < M}. The integrals of |f| over these 
sets must add up to E (|é|). As M increases, one of the integrals increases, while the 
other one decreases. Investigate their limits as M —> oo. 


74 Basic Stochastic Processes 


Thus, for any sequence €, of integrable random variables and any € > 0 
there is a sequence of numbers M, > 0 such that 


/ lEn] dP < €. 
{lEn]> Ma} 


If the Mn are independent of n, then we say that the sequence €, is uniformly 
integrable. 


Definition 4.2 


A sequence €;,£,... of random variables is called uniformly integrable if for 
every € > 0 there exists an M > 0 such that 


| lEn| dP < € 
{l€n|>M} 


for alln =1,2,.... 

Exercise 4.4 

Let N = [0,1] with the o-field of Borel sets and Lebesgue measure. Take 
én = Mio, 2). 

Show that the sequence £é, 2,... is not uniformly integrable. 


Hint What is the integral of En over {En > M} ifn > M? 


Proposition 4.2 


Uniform integrability is a necessary condition for a sequence €1,&2,... of inte- 
grable random variables to converge in L}. 


Lemma 4.2 


If € is integrable, then for every € > 0 there is a 6 > 0 such that 


P(A) <i= | le |dP <e. 
A 


Proof (of Lemma 4.2) 
Let £ > 0. Since € is integrable, by Exercise 4.3 there is an M > 0 such that 


[ [E| dP < -; 
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Now 


/ | dP = J El dP + | £| dP 
A AN{|E| <M} AN{|§|>M} 


< J maps | E| dP 
A {|é|>M} 


< MP(A) + =. 
Let 6 = 547. Then 
P(A) < 8 = | [gla <e, 
A 


as required. 0O 


Exercise 4.5 


Let € be an integrable random variable and Fi, Fo,... a filtration. Show that 
_E(€|F,) is a uniformly integrable martingale. 


Hint Use Lemma 4.2. 


Proof (of Proposition 4.2) 


Suppose that €, > € in L}, i.e. E |En — E| > 0. We take any e > 0. There is an 
integer N such that : 
n > N => Efn -8| < 5- 


By Lemma 4.2 there is a 6 > 0 such that 


P(A) < 5 = | El dP < $. 
A - 


2 


Taking a smaller ô > 0 if necessary, we also have 
P(A) <s= | Enl dP <E forn=1,...,N. 
A 


We claim that there is an M > 0 such that 
P {|En > M} <6 


for all n. Indeed, since 


E (\€nal) > J uy RaldP > MP {fal > M}, 


En |: 


it suffices to take 1 
M = =sup E (||). 
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‘Because the sequence ĉn converges in Lt, it is bounded in L}, so the supremum 


s < œ.) 
Now, since P {|&,| > M} <ô, 


Í EnļdP < é|dP + J én — EldP 
{|En|>M} er. {|En|>M} 


t 7 E (|En — £l) 


lA 
me I L 


ae 


se 


i lEn|dP < € 
{lEn >M} 


or any n = 1,..., N, completing the proof. O 


or any n > N and 


Exercise 4.6 


\how that a uniformly integrable sequence of random variables is bounded in 


i ie. 


sup E (En|) < oo. 


Tint Write E (|€n|) as the sum of the integrals of |En | over {|€,| > M} and {|é,| < M}. 


Exercise 4.6 implies that each uniformly integrable martingale satisfies the 
onditions of Doob’s theorem. Therefore it converges a.s. to an integrable ran- 
lom variable. We shall show that in fact it converges in L+. 


[Theorem 4.3 


ivery uniformly integrable supermartingale (submartingale) En converges in L+}. 


roof 


3y Exercise 4.6 the sequence £n is bounded in L}, so it satisfies the conditions 
f Theorem 4.2 (Doob’s Martingale Convergence Theorem). Therefore, there is 
n integrable random variable € such that En — € a.s. We can assume without 
ss of generality that € = 0 (since €, — € can be taken in place of én). That is 
O say, 


P {lim €n = of =a, 
t follows that €, — 0 in probability, i.e. for any € > 0 


P{lf_l ye} 4 
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as n — oo. This is because by Fatou’s lemma 
lim sup P {|€,| >e} < P (tim sup {|En| > e} ) 


P(2\ {lim &, = of) 
= 0. 


lA 


Let £ > 0. By uniform integrability there is an M > 0 such that 


J énļdP < Ê 
{|En|>M} 3 


for all n. Since én —> 0 in probability, there is an integer N such that ifn > N, 


then 
E 


P {lel > 3} < aay. 


We can assume without loss of generality that M > §. Then 


J En] dP + J lén| dP 
{|€n|>M} {M>|én|>£} 


ad e 
+MP {lén > Z} + EP {5 > él} 


E (|En) 


IA 


2 
3 3 
E 


for all n > N. This proves that E (|€,|) > 0, that is, &, > 0 in LZ}. O 


Theorem 4.4 
Let €, be a uniformly integrable martingale. Then 
En = E (E|Fn) ’ 


where € = limy én is the limit of En in L! and Fn = 0 (&1,...,€n) is the filtration 
generated by én. 


Proof 


For any m >n 


E (Em|Fn) = Én, 


| fmaP= | gar 


i.e. for any A E€ Fn 
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Let n be an arbitrary integer and let A € Fn. For any m >n 





| G9 aP m fén- arl 
JA A 
< f lém -= éldP 
< E(\&m — El) 7 0 


as m — oo. It follows that 


he- | sar 


for any A € Fn, so En = E(E|F,). 


Exercise 4.7 


Show that if én is a martingale and €, — a in L? for some a € R, then 
En = Q a.s. for each n. 


Hint Apply Theorem 4.4. 


Theorem 4.5 (Kolmogorov’s 0-1 Law) 


Let 71,72,... be a sequence of independent random variables. We define the 
tail o-field 
TETN 1s Nea 


where Tn = 0 (Mn, n+1,---). Then 


for any Mpa 


Proof 
Take any A € 7 and define 


P(A) =Oor1 


where Fn = o(m,..-,;1). By Exercise 4.5 En is a uniformly integrable mar- 
tingale, so én — € in L!. By Theorem 4.4 


E (€|Fn) = E (14|Fn) 
for all n. Both € = lim, En and 14 are measurable with respect to the o-field 


T oa: pel wae mM. \ 
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The family G consisting of all sets B € F such that fp EdP = fp ladP is a o- 
field containing 7; UF2U--- . Asa result, G contains the o-field Fæ generated 
by the family Fi U Fo U---. By Lemma 2.1 it follows that € = 14 a.s. 

Since Nn is a sequence of independent random variables, the o-fields Fn 
and 7,4; are independent. Because 7 C 7n41, the o-fields Fan and 7 are 
independent. Being 7-measurable, 14 is therefore independent of Fn for any 
n. This means that 

En = E (14|Fn) = E (14) = P(A) as. 


Therefore the limit liMn—=oo En = € is also constant and equal to P(A) a.s. This 
means that P(A) = 14 a.s., so P(A)=00or1. 0 


Exercise 4.8 


Show that if An € o (En) for each n, then the events 


lim sup Án = N U A; 


j21i>j 
and 
lim inf An = U N Åi 
belong to the tail o-field 7. 
Hint You need to write limsup, An and liminfn An in terms of Ak, Ak+1,... for any 


k, that is, to show that limsup, An and liminfn An will not be affected if any finite 
number of sets are removed from the sequence Aj, Ag,.... 


Exercise 4.9 


Use Kolmogorov’s 0-1 law to show that in a sequence of coin tosses there are 
a.s. infinitely many heads. — 


Hint Show that the event 
{71,72,..- contains infinitely many heads} 


belongs to the o-field 7. Can the probability of this event be 0? The probability of 
the event 
{1,72,.-.. contains infinitely many tails} 


should be the same. Can both probabilities be equal to 0? Can you simultaneously 
have finitely many heads and finitely many tails in the sequence m1, 72,... ? 
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4.4 Solutions 


Solution 4.1 


Because a; = 0 is constant, it is Fo = {0, 2}-measurable. Suppose that ap is 
Fn— -measurable for some n = 1,2,... . Then 

{an =0,E, <a}U{an, =1,&, <b} © F, 
because F,_; C Fn and En is ¥,-measurable. This means that 


On+1 = lfan=0,En <a}U{an=1,En <b} 
is F,-measurable. By induction it follows that a, is Fn—-1-measurable for each 
n = 1,2,... , SO @&1,@?,... is a gambling strategy. 
Solution 4.2 


For a non-negative supermartingale 


sup E (gnl) = sup E (fn) < E(&) = E (|1|) < 0, 


since 
for each n = 1,2,... . Thus Doob’s Martingale Convergence Theorem implies 
that En converges a.s. to an integrable limit. 


Solution 4.3 
Necessity. Suppose that € is integrable. It follows that 


P i|g| < co} = 1. 


The sequence of random variables |€| 1,)¢)5. m} indexed by M = 1, 2,... is mono- 
tone and 


lE lilij >m} W 0 as M =œ 


on the set {|| < co}, i.e. a.s. By the monotone convergence theorem for inte- 


grals 
J El dP \,0 as M - oo. 
{il> M} 


It follows that for every £ > 0 there exists an M > 0 such that 


/ lel dP < €. 
{lel>M) 
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Sufficiency. Take € = 1. There exists an M > 0 such that 


J E| dP <1. 
{I> M} 


Then 
B(\él) = [ £l dP 


J él dP + | El dP 
{|€|>M} {\€|<M} 
1+ MP {\é| < M} 


1+M<o. 


IA A 


Solution 4.4 
For any M > 0 and any n > M we have 
(0, =) = {En > M}, 


J dP = | ndP = 1. 
{En>M} (0,4) 


This means that there is no M > 0 such that for all n 


1 
{En >M} 2 


The sequence €, is not uniformly integrable. 


SO 


Solution 4.5 


In Example 3.4 it was verified that én = E(€|F,) is a martingale. Let € > 0. 
By Lemma 4.2 there is a 6 > 0 such that 


P(A) < i= | e|dP < e. 
A 
By Jensen’s meaai a < E(|E| |Fn) a.s., so 


E(\él) >/E(Enl) > J 


fn] dP > MP{|én| > M}. 
{lénl>M} 


If we take M > E(|€|)/d, then 
P{\&,| > M} < 4. 
Since {|&,| > M} € Fn, it follows that 


J nldP < / E(|e||Fa) dP = J = J&ldP<e, 
{|En|>M} {|€n|>M} {|En|>M} 


proving that E~n = E(f£|F..) is a uniformly integrable sequence. Y>’ 
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Solution 4.6 


Because ĉn is a uniformly integrable sequence, there is an M > 0 such that for 


all n 
J EnldP <1. 
{|En|>M} 


It follows that 


E (\én|) ee / nl dP + f |En] dP 


< 1+ MP {|En| < M} 
< 1+M < œ 


for all n, proving that n is a bounded sequence in L!. 


Solution 4.7 

By Theorem 4.4, En = E (a|Fn) a.s. But E (a|Fn) = a a.s., which proves that 
En =Qa.s. 

Solution 4.8 


Observe that 


lim sup A, = () J A; 


j2k i>) 
for any k. Since 
Ai € a(i) C Tk 
for every i > k, it follows that 


limsup A, = () U A; E Tr 


j2kirj 
for every k. Therefore 
sup A, E T. 
n 
The argument for liminfn An is similar. 
Solution 4.9 


Let 71,72,... be a sequence of coin tosses, i.e. independent random variables 
with values +1, —1 (heads or tails) taken with probability ¿ each. Consider the 
following event: 


A = {m,72,... contains infinitely many heads}. 
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This event belongs to the tail o-field 7 because 
A = lim sup An, 


where 
An = {nn = heads} € o (mm) 


(see Exercise 4.8). Thus, by Kolmogorov’s 0-1 law P(A) = 0 or 1. However, it 
cannot be 0 because the event 


B = {n,72,... contains infinitely many tails} 


has the same probability by symmetry and N = AUB (there must be infinitely 
many heads or infinitely many tails). 
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Markov Chains 


This chapter is concerned with an interesting class of sequences of random 
variables taking values in a finite or countable set, called the state space, and 
satisfying the so-called Markov property. One of the simplest examples is pro- 
vided by a symmetric random walk €, with values in the set of integers Z. If En 
is equal to some í € Z at time n, then in the next time instance n+ 1 it will jump 
either to i + 1, with probability $, or to i — 1, also with probability 4. What 
makes this model interesting is that the value of €,41 at time n + 1 depends 
on the past only through the value at time n. This is the Markov property 
characterizing Markov chains. There are numerous examples of Markov chains, 
with a multitude of applications. 

From the mathematical point of view, Markov chains are both simple and 
difficult. Their definition and basic properties do not involve any complicated 
notions or sophisticated mathematics. Yet, any deeper understanding of Markov 
chains requires quite advanced tools. For example, this is so for problems related 
to the long-time behaviour of Markov processes. In this chapter we shall try 
to maintain a balance between the accessibility of exposition and the depth of 
mathematical results. Various concepts will be introduced. In particular, we 
shall discuss the classification of states and its relevance to the asymptotic 
behaviour of transition probabilities. This will turn out to be closely linked to 
ergodicity and the existence and uniqueness of invariant measures. 
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5.1 First Examples and Definitions 


Example 5.1 


In some homes the use of the telephone can become quite a sensitive issue. 
Suppose that if the phone is free during some period of time, say the nth 
minute, then with probability p, where 0 < p < 1, it will be busy during the 
next minute. If the phone has been busy during the nth minute, it will become 
free during the next minute with probability q, where 0 < q < 1. Assume that 
the phone is free in the Oth minute. We would like to answer the following two 


questions. 

1) What is the probability z, that the telephone will be free in the nth 

minute? 

2) What is limno £n, if it exists? 

Denote by A, the event that the phone is free during the nth minute and let 
By, = Q \ An be its complement, i.e. the event that the phone is busy during 
the nth minute. The conditions of the example give us 
p, (5.1) 
q, (5.2) 
We also assume that P(o) = 1, i.e. zọ = 1. Using this notation, we have 
Zn = P(An). Then the total probability formula, see Exercise 1.10, together 
with (5.1)-(5.2) imply that 

In+1 = P(An41) 

P(An+i|An)P(An) + P(An+i|Bn)P(Bn) 
= (1-p)tn+q(1-—2n2)=¢+(1-p- Q)an. (5.3) 


P(Bn+i|An) 
P(An+1|Bn) 


| 


It’s a bit tricky to find an explicit formula for £n. To do so we suppose first 
that the sequence {z,,} is convergent, i.e. | 


lim Zp = 2. (5.4) 
n—> OO 


The elementary properties of limits and equation (5.3), ie. Zng1 = q+(1—p-— 


q)£n, yield 
r=qt+(l—p-q)z. (5.5) 


The unique solution to the last equation is 


q 
r = ——. 5.6 
a (5.6) 


NNE SIGUA EEE ENANAR ore ORARAA AE AEH unana aanre 
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In particular, 


q q 
q+p á EF Ce 
Subtracting (5.7) from (5.3), we infer that 
q q | 
n a ee oe gee = 1 m ap n a Se ea e 5.8 
Us” iia oP a) (2 ia Pa 


Thus, {£n — rea is a geometric sequence and therefore, for all n € N, 


ty ~ 2 = (1-p—4)" (wo 4). 


qtp q+p 


Hence, by taking into account the initial condition zọ = 1, we have 


q q n 
Ln = — + | To — — l1—p-gq 
q+p (zo L) ) 
q P 
= ——— + ——(]-—y- gq)". 5.9 
qt+p ie P q) ( ) 


Let us point out that although we have used the assumption (5.4) to derive 
(5.8), the proof of the latter is now complete. Indeed, having proven (5.9), we 
can show that the assumption (5.4) is indeed satisfied. This is because the 
conditions 0 < p,q < 1 imply that |1 — p — q| < 1, and so (1 — p — q)” > 0 as 
n — oo. Thus, (5.4) holds. This provides an answer to the second part of the 


example, i.e. limyn+o Tn = T 


The following exercise is a modification of the last example. 


Exercise 5.1 


In the framework of Example 5.1, let yn denote the probability that the tele- 
phone is busy in the nth minute. Supposing that yọ = 1, find an explicit formula 
for yn and, if it exists, liMn—=oo Yn. 

Hint This exercise can be solved directly by repeating the above argument, or indi- 


rectly by using some of the results in Example 5.1. 


Remark 5.1 


The formulae (5.3) and (5.64) can be written collectively in a compact form by 
using vector and matrix notation. First of all, since £n + yn = 1, we get 


Intl = (1 = p)En + Yn, 
Yn+1 = pln +(1— q)yn. 
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Hence, the matrix version takes the form 
Peet aes q z 
Yn+1 P beg Yn 
The situation described in Example 5.1 is quite typical. Often the proba- 
bility of a certain event at time n + 1 depends only on what happens at time 


n, but not further into the past. Example 5.1 provides us with a simple case of 
a Markov chain. See also the following definition and exercises. 


Definition 5.1 


Suppose that S is a finite or a countable set. Suppose also that a probability 
space (92, F, P) is given. An S-valued sequence of random variables én, n € N, 
is called an S-valued Markov chain or a Markov chain on S if for all n € N and 
alls ES 
P(En+1 = 8|€o,---+€n) = PlEn+i = sén). (5.10) 
Here P(€ni1 = s|€n) is the conditional probability of the event {€,41 = s} with 
respect to random variable én, or equivalently, with respect to the o-field o (€n) 
generated by n. Similarly, P(€n41 = s|€o,.--,&n) is the conditional probability 
of {€,41 = s} with respect to the o-field o(&,---,&n)) generated by the random 
variables éo,- , En. 
Property (5.10) will usually be referred to as the Markov property of the 
Markov chain én, n € N. The set S is called the state space and the elements 


of S are called states. 


Proposition 5.1 


The model in Example 5.1 and Exercise 5.1 is a Markov chain. 


Proof 


Let S = {0,1}, where 0 and 1 represent the states of the phone being free or 
busy. First we need to construct an appropriate probability space. Let N be 
the set of all sequences wo,w 1,... with values in S. Let po be any probability 
measure on S. For example, u = ĝo corresponds to the case when the phone 
is free at time 0. We shall define P by induction. For any S-valued sequence 
S0, S1;,... we put 

P({w € 2: wo = so}) = pol{so}) (5.11) 
and 


Dibre Ferne SS Be a SH ke VE VY) (5.12) 
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as P(Sn41|8n)P({w EN: Wi = Si, 1 = 0, week n}), 
where p(sjr) are the entries of the 2 x 2 matrix 
par p(0|1) | _ | l-p q | 
p(1j0)  p(1|1) Dp henge | 

It seems reasonable to expect P to be a probability measure (with respect to ` 
the trivial o-field of all subsets of 92). Take this for granted, check only that 
P( =. 

How would you define the process ĉn, n € N? We shall do it in the standard 
way, 1.e. 


En(W) =Wn, we 22. (5.13) 
First we shall show that the transition probabilities of €n are what they should 
be, 1.e. 
Plénti = lén =0) =p, (5.14) 
P(En+1 = Of, = 1) = q. (5.15) 
The definition of conditional probability yields 
P(En+i = 1, En = 9) 
PGi S10) = ee 
Pin = 0) 
Next, the definition of P gives 
Peasy = l, En PR 0) 
P({w E N : wn = 0,Wn41 = 1}) 
` PUUE 2:0; = 8;,4=0,---,n—1, wp = onr = 1}) 


S0, Sn-1ES 

E T eee 
S0, ,Sn-1ES 

= PP(En = 0), 


by (5.12). We have proven (5.14). Moreover, (5.15) follows by the same argu- 
ment. A similar line of reasoning shows that €, is indeed a Markov chain. For 


this we need to verify that for any n € N and any s9,81,---,5n41 ES 
O P(En41 = Sny Eo = $0, t, En = Sn) = P (Enti = Sntil€n = Sn). 
We have 
P(£o = 80,'t t, En = Sn, En+1 = Sn+1) 


= P({w € 2:0; = $i,i=0,---,n + 1)}) 
P(sn41|$n)P({w € 2: wi | = 0,--+,n}) 
§ 


p(8n+1|Sn)P (Eo = 80,°°'* En = Sn) 


l 
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which, in view of the definition of conditional probability, gives 


P(En+1 = $n41/60 = 805°+7)€n = Sn) = P(Sn41|8n).- 


On the other hand, by an easy generalization of (5.14) and (5.15) 
P(Enti = Sn4il€n = $n) = P(Sn4+1|8n); 
which proves (5.10). 0 


In Example 5.1 the transition probabilities from state 2 to state 7 do not 
depend on the time n. This is an important class of Markov chains. 


Definition 5.2 


An S-valued Markov chain én, n € N, is called tzme-homogeneous or homoge- 
neous if for all n € N and all i,j E€ S 


P(En+1 = Jln = i) = P(E: = jléo = 2). (5.16) 


The number P(&1 = j|&) = i) is denoted by p(j|z) and called the transition 
probability from state i to state j. The matrix P = [p(j|t)], ;-5 is called the 
transition matrix of the chain £n. 


Exercise 5.2 


In the discussion so far we have seen an example of a transition matrix, P = 

| lap q 
po tg 

to 1. Prove that this is true in general. 


| . Obviously the sum of the entries in each column of P is equal 


Hint Remember that P(2|A) = 1 for any event A. 


Definition 5.3 
A = [aşili jes is called a stochastic matriz if 

1) aji > 0, for all i,j € S; 

2) the sum of the entries in each column is 1, i.e. jes aji = 1 for any i E€ S. 


A is called a double stochastic matriz if both A and its transpose At are stochas- 
tic matrices. 
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Proposition 5.2 


Show that a stochastic matrix is doubly stochastic if and only if the sum of the 
entries in each row is 1, i.e. J jcs aji = 1 for any j E S. 


Proof 


Put A’ = [b,;]. Then, by the definition of the transposed matrix, bj; = aji. 
Therefore, A’ is a stochastic matrix if and only if 


dai = D big = 1, 
completing the proof. O 


Exercise 5.3 


Show that if P = [p;i]; ies is a stochastic matrix, then any natural power P” of 
P is astochastic matrix. Is the corresponding result true for a double stochastic 
matrix? 


Hint Show that if A and B are two stochastic matrices, then so is BA. For the second 
problem, recall that (BA) = A‘ B®. 


Exercise 5.4 


Let P=| 75? q |; Show that 
p l—q 


p? — | l+p*—2p+pq 2q-pq-q 
2p- pq-p? 1+q*—2q¢+pq 


Hint This is just simple matrix multiplication. 


We see that there is a problem with finding higher powers of the matrix P. 
When multiplying P* by P, P?, and so on, we obtain more and more compli- 
cated expressions. 


Definition 5.4 
= The n-step transition matriz of a Markov chain En with transition probabilities 
p(jli), j,i € S is the matrix P, with entries | 


Pal) = Plin = jlo = i). (5.17) 
/ 
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Exercise 5.5 


Find an exact formula for P, for the matrix P from Exercise 5.4. 


Hint Put tn = P(n = 0|fo = 0) and yn = P(En = 1|€o = 1). Is it correct to suppose 
that pn(0|0) = £n and pn(1|1) = yn? If yes, you may be able use Example 5.1 and 
Exercise 5.1. 


Exercise 5.6 


You may suspect that P, equals P”, the nth power of the matrix P. This holds 
for n = 1. Check if it is true for n = 2. If this is the case, try to prove that 
P, = P” forallneN. 


Hint Once again, this is an exercise in matrix multiplication. 


The following is a generalization of Exercise 5.6. 


Proposition 5.3 (Chapman—Kolmogorov equation) 


Suppose that én, n € N, is an S-valued Markov chain with n-step transition 
probabilities pn (jli). Then for all k,n € N 


Pr+k(jlt) = > _ Pn(J|s)pa(sli), ij € S. (5.18) 
sES 


Exercise 5.7 


Prove Proposition 5.3. 


Hint pn+k(jli) are the entries of the matrix P,4, = Pt’. 


Proof (of Proposition 5.3) 


Let P and FP, be, respectively, the transition probability matrix and the n-step 
transition probability matrix. Since pn(j|z) are the entries of P,, we only need 
to show that P, = P” for all n € N. This can be done by induction. The 
assertion is clearly true for n = 1. Suppose that Pa = P”. Then, for i,j E€ S, 
by the total probability formula and the Markov property (5.10) 


Pn+1(3|t) = PEn+1 = J |Eo = i) 


— S P(Enti = j|&o = we = SIP CEA — sl& = i) 
sES 
ig > PGs = Wea = sP En = s|£o = 1) 


oo C 
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= X p(i|s)pn (sla), 


sES 


which proves that Pa+ı = PPa. O 


Exercise 5.8 (random walk) 


Suppose that S = Z. Let nn, n > 1 be a sequence of independent identically 
distributed random variables with P(ņ = 1) = p and P(m, = -1) =q =1-p. 
Define £, = J; ni for n > 1 and & = 0. Show that €, is a Markov chain 
with transition probabilities 


D, if j=i+l1, 
pll) =$ q, if g=i-], 
0, otherwise. 


En, n > 0, is called a random walk starting at 0. Replacing éo = 0 with éo = t, 
we get a random walk starting at i. 


Hint En+1 = En + Mn+1. Are En and 7,41 independent? 


Exercise 5.9 


For the random walk €, defined in Exercise 5.8 prove that 
: ; n n+j—-i n—jt+i 
Pls = jle =) = (nips J E (5.19) 
2 
ifn +j-— iis an even non-negative integer, and P(n = 7|€) = i) = 0 otherwise. 


n+j-i 
Hint Use induction. Note that ( n+p )p 2 equals 0 if |7 —7] >n+1. 


Proposition 5.4 


For all p € (0,1) 
P(E, = i/o =i) 70, as n > oo. (5.20) 


Proof 


To begin with, we shall consider the case p # $. When j = i, formula (5.19) 
becomes (an)! 
š if n= 2k 
Ple =ilé& <i) =) ©? (pq)", i , a 
(E Ifo = 0) 0, | if n is odd. ey 
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Then, denoting ak = faye (pq)*, we have 


arpi (2k +1)(2k +2) 
ak m (k +1) 





—+ 4pq < 1. 


Hence, ap — 0. Thus, P(ĉ& = il = i) —> 0. The result follows, since 
P(£2k+1 = ilfo = t) = 0 > 0. 

This argument does not work for p = $ because 4pq = 1. In this case we 
shall need the Stirling formula! 


PN 
k! ~ V2rk (£) , as k > œ. (5.22) 


Here we use the standard convention: an ~ bn whenever + — 1 as n > ov. 
By (5.22) 





Aak ~N 


al eJ (<)" (pq)* 


27k e k 


1 
= —— > 0, as k~ œ. 


Vrk 


Let us note that the second method works in the first case too. However, in the 
first case there is no need for anything as sophisticated as the Stirling formula. 
O 


Proposition 5.5 


The probability that the random walk €, ever returns to the starting point is 


l1- |p- ql. 


Proof 


Suppose that & = 0 and denote by fo(n) the probability that the process 
returns to 0 at time n for the first time, i.e. 


foln) = P (En =0,& #0, = 1, n — 1). 


If also po(n) = P (En = 0) for any n € N, then we can prove that 


OO OO 


> po(n) = X poln) > fo(n). (5.23) 


n=0 n=l 


! See, for example, E.C. Titchmarsh, The Theory of Functions, Oxford University 


Danann L^\--fnn A TWO 
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Since all the numbers involved are non-negative, in order to prove (5.23) we 
need only to show that 


n) = $ fo(k)po(n — k) forn > 1. 
k=1 
The total probability formula and the Markov property (5.10) yield 


poln) = Sor ere Oe = 0,6 #40,7=1,---,k-1) 
kæli 
= 2 Pesii 70S eske 
k=1 
xP(En = Of = 0,& #0,i = 1, -,k — 1) 


= Ý P(& =0,& #0,i = 1, k — 1)P(En = 01& = 0) 


k=1 


= 3 fo(k)po(n — k). 
k=1 


Having proved (5.23), we are going to make use of it. First we notice that the 
probability that the process will ever return to 0 equals $7 fo(n). Next, from 
(5.23) we infer that 


P(3n >1:&,=0) = $ fo(n) 


=1- (Em) =1- (Eme) 
n=0 k=0 


Since po(2k) = rae (pg)* and 


aaa ate (1 — 42)? jal < =, (5.24) 


it follows, that for p Æ 1/2 
P(In > 1: &, = 0) = 1- (1 — 4p)? = 1- |p- ql, (5.25) 


since, recalling that q = 1 — p, we have 1 — 4pq = 1 — 4p + 4p* = (1 — 2p)? = 
(q — p)’. 

The case p = 1/2 is more delicate and we shall not pursue this topic here. 
Let us only remark that the case p = 1/2 needs a special treatment as in 
Proposition 5.4. O 
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Exercise 5.10 
Prove formula (5.24). 


Hint Use the Taylor formula to expand the right-hand side of (5.24) into a power 
series. 


Exercise 5.11 (branching process) 


On the island Elschenbieden there lives an almost extinct species called Vugiel. 
Vugiel’s males can produce zero, one, two, --- male offspring with probability 
Po, Pi, P2,° +: respectively, where p; > 0 and X` ;<o Pi = 1. A challenging problem 
would be to find the Vugiel’s chances of survival assuming that each individual 
lives exactly one year. At this moment, we ask you only to rewrite the problem 
in the language of Markov chains. 


Hint The number of descendants of each male has the same distribution. 


Exercise 5.12 


Consider the following two cases: 


1) In Exercise 5.11 suppose that 


Pm = ("oma -p^ 


for some p € (0,1) and N € N*, where N* = {1,2,3,---}. (Note that 
Pm = 0 if m > N.) Show that 


pili) = (F) pa -pn (5.26) 


Deduce that, in particular, p(j|z) = 0 if j > Ni. 


2) Suppose that 
A 
Pm Ss | eo) m € N, 
m 


. 


for some A > 0. In other words, assume that each X; has the Poisson 
distribution with mean 4. Show that 


E. NET 
pli) = “Re, 5,120. (5.27) 





Hint If X, has the binomial distribution P(X; = m) = (")p™(1— p)*-™, m E€ N, 
then there exists a finite sequence n}, ---, nọ of independent identically distributed 
random variables such that P(n} = 1) = p. P(n? = 0) = 1 — p and X; =m +- + 
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Th = = m. Hence, X1 +---+ X; =} N —; 7s, i.e. the sum of Ni independent 


r=1 
identically distributed sander variables with distribution as above. Hence we infer 
(5.26). 


Proposition 5.6 


The probability of survival in Exercise 5.12, part 2) equals 0 if A < 1, and 1—7* 
if A > 1, where k is the initial Vugiel population and fr € (0,1) is a solution to 


p= eTA, (5.28) 


Proof 


We denote by ¢(z), i € N the probability of dying out subject to the condition 
Eo = i. Hence, if A = {£, = 0 for n € N}, then 


pli) = P (Alf = i). (5.29) 


Obviously, ¢(0) = 1 and the total probability formula together with the Markov 
property (5.10) imply that for each i € N 


y PAG=(42=)PG=ia=9 


j=0 


SP ( (Ali = 7) P (& = go = 2) 


7=0 


S o(3)p(3li). 
7=0 


Therefore, the sequence ¢(2), i € N is bounded (by 1 from above and by 0 from 
below) and satisfies the following system of equations 


p(t) 


pi) = J PGi, TEN (5.30) 
(0) = 1. 


So far, we have not used any particular distribution of X;j. From now on, we 
shall assume that the X; have the Poisson distribution. Hence, by Exercise 5.12, 
pG) = ay e—** It is not an easy problem to find a solution to (5.30), even in 
this petal case. ġ(i) is the probability that the population will die out, subject 
to the condition that initially there were t individuals. Since we assume that 
reproduction of different individuals is independent, it is reasonable to make 


the following Ansatz: | 
Ali) = Alh. i EN. (5.31) 
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for some A > 0. Although it is possible to prove this Ansatz, we shall not 
do so here. Note that the boundary condition ¢(0) = 1 implies that A = 1. 
Substituting (5.31) (with A = 1 and r := ¢(1)) into (5.30), we get 


CO . 
(A)? in 
t- ak J 2 
r = ) r 7 e 


j=0 
— J 
zg 5 = (ird) See 
j=0 7 


Hence, r should satisfy 

r =ef, (5.32) 
Since the function g(r) = ei) re [0,1], is convex, there exist at most 
two solutions to the equation (5.32). Obviously, one of them is r = 1. A bit of 
analysis, not included here, shows the following: 


1) If \ < 1, then the only solution to (5.32) in [0,1] is r = 1. 


2) If A > 1, then there exists a second solution f € (0,1) of the equation 
(5.32). œ 


In case 1) the situation is simple. We have ¢(7) = 1 for all 7, and thus the 
probability of extinction is 1 for any initial number of individuals. Case 2) is 
slightly more involved. The first question we need to address is which of the 
two solutions of (5.32) gives the correct value of ¢(1)? Recall that p, = Xen, 
Define 


OO 


P(e) = X pir" = =y. Xe a a nA Ea (5.33) 
k=0 


k=0 
Since P(€,; = 0/&> = 1) = po and 


P (& =O = 1) = X P (& = 06 = i) P (& = ilfo = 1) 


1-0 


= > (po)'pi = F (po) = F(F(0)), 


1=0 
we guess that the following holds: 
P(En = Ol = 1) = F™ (0), (5.34) 


where F'”) is the n-fold composition of F. To prove (5.34) it is enough to prove 
it for n, while assuming it holds for n — 1. We have 


P (En = 06 =1) = S>P(&, = 0/6 =i) P(& =| =1) 





Ee 
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= Y pP (En-1 = Olfo = i) 


1=0 


= Spi [P(Er-1 = Olf0 = 1)] 
i=0 


= Yn: [FeO] = FERPO) = P™O) 


Since the event {én = 0} is contained in events {€,41 = 0} for all n € N, we 
have 


| 


P {€, = 0, for some n € Nf = 1} 
jim, P {én = O|fo = 1} 


(1) 


by the Lebesgue monotone convergence theorem. Therefore, we infer that 
é(1) = lim F™ (0). 

With F(z) = x we only need to show that 
F'(0) <f, nen. (5.35) 


Indeed, once the inequality (5.35) is proven, we infer that ¢(1) < f and thus 
@(1) = f. We shall prove (5.35) by induction. It is obviously valid for n = 0, so 
we need to study the inductive step. We have 


F™ (0) = F (F"-(0)) < F(r) =f, 


since F' is increasing. We conclude that in the case A > 1 the population will 
become extinct with positive probability. 

In the simplest example of the binomial distribution case, i.e. when N = 1, 
equations (5.30) become 


6) = 2 00)(")ra py, ie N. 
oÇ) 
Since (0) = 1, ¢(1) satisfies 

(1) =q + 4(1)p 


with q = 1 — p. Hence, trivially, 6(1) = 1. Then, by induction, one proves that 
(i) = 1. Therefore, whatever the initial number of individuals, extinction of 
the species is certain. O 
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Remark 5.2 


The method presented in the last solution works for any distribution of the 
variables X,. It turns out that the mean value A of Xı plays the same role as 
above. One can show that if A < 1, then the population will become extinct 
with probability 1, while for A > 1 the probability of extinction is larger than 
O and smaller than 1. 


Exercise 5.13 


On the, now familiar, island of Elschenbieden the question of survival of the 
Vugiel is a hot political issue. The (human) population of the island is N. Those 
who believe that action should be taken in order to help the animals preach 


their conviction quite convincingly. For if a supporter discusses the issue with a- 


non-supporter, the latter will change his mind with probability one. However, 
they do so only in face-to-face encounters. Suppose that the probability of an 
encounter of exactly one pair of humans during one day is p and that with 
probability q this pair is a supporter—non-supporter one. Write down a Markov 
chain model of this situation. Neglect the probability of two or more encounters 
during one day. 


Hint On each day the number of supporters can either increase by 1 or remain un- 
changed. What is the probability of the former? 


Exercise 5.14 (queuing model) 


A car wash machine can serve at most one customer at a time. With probability 
p, 0 < p< 1, the machine can finish serving a customer in a unit time. If this 
happens, the next waiting car (if any) can be served at the beginning of the 
next unit of time. During the time interval between the nth and (n + 1)st unit 
of time the number of cars arriving has the Poisson distribution with parameter 
A > 0. Let €, denote the number of cars being served or waiting to be served 
at the beginning of unit n. Show that En, n € N, is a Markov chain and find 
its transition probabilities. 


Hint Let Zn, n = 0,1,2,:-- be a sequence of independent identically distributed 
random variables, each having the Poisson distribution with parameter A. Then €n41— 
En — Zn equals —1 or 0. 


Remark 5.3 


In the last model we are interested in the behaviour of €, for large values of n. 
In particular, it is interesting to determine whether the limit of €, or that of Eé,, 
(as n = oo) exists. In Exercise 5.36 we shall find canditianns which onarantee 
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the existence of a unique invariant measure and imply that the Markov chain 
in question is ergodic. 


5.2 Classification of States 


In what follows we fix an S-valued Markov chain with transition matrix P = 
[p(jli)]; ics, where S is a non-empty and at most countable set. 


Definition 5.5 


A state i is called recurrent if the process €, will eventually return to 2 given 
that it starts at 2, i.e. 


P(E, = i for some n > 1|€) = 7) = 1. (5.36) 


If the condition (5.36) is not satisfied, then the state z is called transient. 


Theorem 5.1 


Show that for a random walk on Z with parameter p € (0,1), the state 0 is 
recurrent if and only if p = 1/2. Show that the same holds if 0 is replaced by 
any other state îi € Z. 


Proof 


We know from (5.25) that P(€, = i for some n > 1/& = i) = 1 — |p — q| for 
anyic 2.0 


Definition 5.6 


We say that a state 1 communicates with a state j if with positive probability 
the chain will visit the state 7 having started at i, i.e. 


P(E, = j for some n > O|f = i) > 0. (5.37) 


If 2 communicates with j, then we shall write 1 — 7. We say that the state i 
intercommunicates with a state j, and write 2 4 7, if i 4 j and j => i. 


Fxercise 5.15 
Show that i > 7 if and only if p,(j|t) > 0 for some k > 1. 
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Hint Recall that p (jli) = P(€ = jl€o = i). 

Exercise 5.16 

Show that 

1) ii, 

2) if ¿ © J then 7 © i and 

3) ifi j, j e ktheni e k. 
In other words, show that + is an equivalence relation on S. 


Hint 1) and 2) are obvious. For 3) use the Chapman-Kolmogorov equations. 


Exercise 5.17 
For |z| < 1 and 7,2 € S define 


Pyi(z) = $ pr(jli)a”, (5.38) 


n=0 


$ faldlie”, (5.39) 


s 
Ə 
i 


where fn(j|t) = P (En = J Ek £ j,k =1,---,n—1]& = i). Show that the power 
series in (5.38)-(5.39) are absolutely convergent for |z| < 1 and that 
Palæ) = Fyi(w)Pij(a), ifj £i, (5.40) 


Hint Note that |pn(jli)| < 1, so the radius of convergence of the power series (5.38) 
is > 1. 


Exercise 5.18 
Show that limg a1 Pj;() = P po Pn (j|) and limg a1 Fj (£) = Dopo fn (ili). 


Hint Apply Abel’s lemma’: If a, > 0 for all k > 0 and lim sup} */lax| < 1, then 
lims 71 go apr? = > po 4k, HO matter whether this sum is finite or infinite. 


* For example, see: W. Rudin, Principles of Mathematical Analysis, McGraw-Hill 
Book Company, New York 1976. 
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Exercise 5.19 


Show that a state j is recurrent if and only if $`, pn(j|j) = oo. Deduce that 
the state j is transient if and only if 


> Palili) < œ. (5.42) 


Show that if 7 is transient, then for each i € S 


> Pn(ili) < œ. (5.43) 


Hint If j is recurrent, then Fj;(x) + )) fa(jlj) = 1 asx 1. Use (5.41) in 
conjunction with Abel’s Lemma. 


Exercise 5.20 


l l— 
For a Markov chain £, with transition matrix P = | 5 p i | show that 
-q 


both states are recurrent. 


Hint Use Exercise 5.19 and 5.5. 


One may suspect that if the state space S is finite, then there must exist 
at least one recurrent state. For otherwise, if all states were transient and 
S = {1,2,---,N}, then with positive probability a chain starting from 1 would 
visit 1 only a finite number of times. Thus, after visiting that state for the 
last time, the chain would move to a different state, say ig, in which it would 
stay for a finite time only with positive probability. Thus, in finite time, with 
positive probability, the chain will never return to states 1 and 22. By induction, 
in finite time, with positive probability, the chain will never return to any of 
the states. This is impossible. The following exercise will give precision to this 
argument. : 


Exercise 5.21 


Show that if é» is a Markov chain with finite state space S, then there exists 
at least one recurrent state i € S. 


Hint Argue by contradiction and use (5.43). 


The following result is quoted here for reference. The proof is surprisingly 
difficult and falls beyond the scope of this book. 
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Theorem 5.2 


A state 7 € S is recurrent if and only if 
P(€, = 7 for infinitely many n|& = j) = 1, 
and is transient if and only if 


P(€, = j for infinitely many n|€ = 7) = 0. 


Definition 5.7 


For an S-valued Markov chain £n, n € N, a state i € S is called null-recurrent 
if it is recurrent and its mean recurrence time m, defined by 


OO 


mi := $ nfalili) (5.44) 


n=0 


equals oo. A state i € S is called positive-recurrent if it is recurrent and its 
mean recurrence time m; is finite. 


Remark 5.4 


One can show that a recurrent state i is null-recurrent if and only if pn (ili) — 0. 


We already know that for a random walk on Z the state 0 is recurrent if and 
only if p = 1/2, i.e. if and only if the random walk is symmetric. In the following 
problem we shall try to answer if 0 is a null-recurrent or positive-recurrent state 
(when p = 1/2). 

Exercise 5.22 


Consider a symmetric random walk on Z. Show that 0 is a null-recurrent state. 
Can you deduce whether other states are positive-recurrent or null-recurrent? 


Hint State 0 is null-recurrent if and only if )) nfn(0|0) = oo. As in Exercise 5.18, 
>), nfa(0|0) = lim, »1 Foo(x), where Foo is defined by (5.39). i 


Exercise 5.23 


For the Markov chain €, from Exercise 5.20 show that not only are all states 
recurrent, but they are positive-recurrent. 


Hint Calculate f,(0|0) directly. 
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The last two exercises suggest that the type of a state i € S, i.e. whether 
it is transient, null-recurrent or positive-recurrent is invariant under the equiv- 
alence &. We shall investigate this question in more detail below, but even 
before doing so we need one more notion: that of a periodic state. 


Definition 5.8 


Suppose that En, n € N, is a Markov chain on a state space S. Let i € S. We 
say that 7 is a periodic state if and only if the greatest common divisor (gcd) 
of all n € N*, where N* = {1,2,3,---}, such that p,(z|2) > 0 is > 2. Otherwise, 
the state i is called aperiodic. In both cases, the gcd is denoted by d(z) and is 
called the period of the state 7. Thus, 7 is periodic if and only if d(z) > 2. A 
state 2 which is positive recurrent and aperiodic is called ergodic. 


Exercise 5.24 


Is this claim that pa) (ili) > 0 true or not? 


Hint Think of a Markov chain in which it is possible to return to the starting point 
by two different routes. One route with four steps, the other one with six steps. 


One of the by-products of the following exercise is another example of the 
type asked for in Exercise 5.24. 


Exercise 5.25 


Consider a Markov chain on S = {1,2} with transition probability matrix 

—{ 0 1/2 | 
os | 1 1/2 
Find d(1) and d(2). 


| . This chain can also be described by the graph in Figure 5.1. 


1/2 


Figure 5.1. Transition probabilities of the 


1/2 Markov chain in Exercise 5.25 


Hint Calculate P? and P*. This can be done in two different ways: either algebraically 
or, probabilistically. 
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Suppose that i,j € S andi © j. Show that 
1) i is transient if and only if 7 is; 
2) zis recurrent if and only if 7 is; 
3) iis null-recurrent if and only if 7 is; 
4) i is positive-recurrent if and only if j is; 
5) i is periodic if and only if j is, in which case d(i) = d(j); 


6) 7 is ergodic if and only if 7 is. 


Proof 


It is enough to show properties 1), 4) and 5). Since i + j one can find n,m € N 
such that pm(j|t) > 0 and pn(ilj) > 0. Hence € := pm(jli)pn(ilj) is positive. 
Let us take k € N. Then by the Chapman—Kolmogorov equations 


Pm+k+nljlj) = >> Pm(Jls)pe(slr)pn(rli) > Pm(jli)pr (tlt) Pa (tld) = epr (ili). 
r,sES 


By symmetry 


Pnyk+m(ili) = X pnlils)pr(slr)Pm(rli) > pr(ilj)pr (lj) Pm (ili) = epe (Jd). 
r,sES 


Hence, the series >, px (il?) and >>, pk(j|j) are simultaneously convergent or 
divergent. Hence 1) follows in view of Exercise 5.19. 
To prove 5) it is enough to show that 


d(i) < d(j). 
Using the first inequality derived above, we have 
Pn+k+m (ili) > epr (jli) 
for all k € N. From this inequality we can draw two conclusions: 
(a) d(i)|n + m, since by taking k = 0 we get Pn+m(ili) > 0; 


(b) if pe (Jlj) > 0, then Pn+r+m(ili) > 0. 
From (a) and (b) we can see that d(i)|k provided that p,(j|j) > 0. This proves 
what is required. O 
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Exercise 5.26 


Show that the following modification of 2) above is true: If i is recurrent and 
i + j, then j —> i. Deduce that if 7 is recurrent and 2 — j, then 7 is recurrent 
and 7 + 1. 


Hint Is it possible for a chain starting from 2 to visit 7 and then never return to 2? 
Is such a situation possible when 2 is a recurrent state? 


The following result describes how the state space S can be partitioned into 
a countable sum of classes. One of these classes consists of all transient states. 
Each of the other class consists of interconnecting recurrent states. If the chain 
enters one of the classes of second type, it will never leave it. However, if the 
chain enters the class of transient states, it will eventually leave it (and so never 
return to it). We begin with a definition. 


Definition 5.9 


Suppose that En, n € N, is a Markov chain on a countable state space S. 


1) A set C C S is called closed if once the chain enters C' it will never leave 
it, 1.e. 
P(& €S\C for some k > nl&, E C) = 0. (5.45) 


2) A set C C S is called irreducible if any two elements i,j of C intercommu- 
nicate, i.e. for all i,j € C there exists an n € N such that pn(jli) > 0. 


Theorem 5.3 
Suppose that én, n € N, is a Markov chain on a countable state space S. Then 
N 
S=TU\|JC;, (disjoint sum), (5.46) 
j=l 


where T is the set of all transient states in S and each C; is a closed irreducible 
set of recurrent states. 


Exercise 5.27 


Suppose that én, n € N, is a Markov chain on a countable state space S. Show 
that a set C C S is closed if and only if p(j|z) = 0 for allie CandjeS\C. 


Hint One implication is trivial. For the other one use the countable additivity of the 
measure P. 
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Proof (of Theorem 5.3) 


Let R = S \ T denote the set of all recurrent states. If i + j, then both 2 
and j belong either to T or to R. It follows that the interconnection relation 
+ restricted to R is an equivalence relation as well. Therefore, R = Jo Cis 
C; = [s;], s; E R. Here N denotes the number of different equivalent classes. 
Since by definition each C} is an irreducible set, we only need to show that it 
is closed. But this follows from Exercise 5.26. Indeed, if i € Ck and i — 7, then 
i + j, and so j € C. O 


5.3 Long-Time Behaviour of Markov Chains: 
General Case 


For convenience we shall denote the countable state space S by {1,2,3,---} 
when S is an infinite set and by {1,2,---,n} when S is finite. 


Proposition 5.8 


Let P = [p(j|t)] be the transition matrix of a Markov chain with state space S. 
Suppose that for all i,j € S 


lim Pr (Jlt) eee (5.47) 
(In particular, the limit is independent of 7.) Then 
1) 5 Tj <l; 


2) Ja PUli)m = Tj; 
3) either )), t; = 1, or 7j = 0 for all j € S. 


Proof 


To begin with, let us assume that S is finite with m elements. Using the 
Chapman-Kolmogorov equations (5.18), we have 


m m 

> m =} m=} lim p(l) 

JES j=l j=1 
m 

lim J Pa(jli) = lim 1=1, 
j=1 


n— o0 
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since A Pn(j|t) = 1 for any n € N (see Exercise 5.2). This proves 1) and 3) 
simultaneously. Moreover, it shows that the second alternative in 3) can never 
occur. To prove 2) we argue in a similar way. Let us fix j € S and (an auxiliary) 
k €e S. Then, 


> pGlijmi = $ lim p(jli)pn(ilk) 
i=l i=l 


lim > P(li)pa(ilk) = lim pn+ı(j|k) = Tj, 


Il 


since `; p(jli)pn (ilk) = pn4i(j|k) by the Chapman—Kolmogorov equations. 

When the set S is infinite, we cannot just repeat the above argument. The 
reason is quite simple: in general the two operations lim and $` cannot be 
interchanged. They can when the sum is finite, and we used this fact above. 
But if S is infinite, then the situation is more subtle. One possible solution of 
the difficulty is contained in the following version of the Fatou lemma. 


Lemma 5.1 (Fatou) 
Suppose that a;(n) > 0 for 7,n € N. Then 


2 lim inf a;(n) < lim inf 2 aj(n (5.48) 


Moreover, if aj(n) < 6; for j,n € N and `}; bj < œ, then 


lim sup ` aj(n) < ` lim sup a; (n). (5.49) 
n j j n 


Using the fact that for a convergent sequence lim and lim inf coincide, by 
the Fatou lemma we have 


D i 


JES 


OO 
= dad im pn(jli) 


eet Pn = lim inf 1 =]! 
J= 


1e 


IA 


since, as before, Da pn(jli) = 1 for any n € N. This proves 1). A similar 
argument shows 2). Indeed, with j € S and k € S fixed, by the Chapman- 
Kolmogorov equations and the Fatou lemma we have 


> pll: = X lim p(li)pn(ilk) 
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< lim aint > p(lipa(ilk) = = lim n inf pn+1(J |k) = Tj? 


i=] 


To complete the proof of 2) suppose that for some k € S 


X plkļi)ri < Tk. 


(=l 


Then, since ĵ jes Tj = J jgk Tj + Tk, by the part of 2) already proven we have 


a > > (Dr T + Lop (kli) 7 


j=l jÆk i 
=>, dP (limi = 5 Spo 
j=1 i=l j 


iM 3 


- En (Frun) 5 yr 


We used the fact that = p(j|t) = 1 together with (5.65). This contradiction 
proves 2). 
In order to verify 3) observe that by iterating 2) we obtain 


$ Palslé)me = Tj. 
i 


Hence, 


Tj = lim X pnr(jli)ri 
i 
>, lim prljji)r 2 TiN; = Tj dt. 
1 4 1 


Therefore, the product 7; (>), 7; — 1) is equal to 0 for all j € S. As a result, 3) 
follows. Indeed, if $); m; 4.1, then 7; = 0 for all j € S. D 


Definition 5.10 


A probability measure u := )/..¢ Hjĝj is an invariant measure of a Markov 
chain €,, n € N, with transition probability matrix P = [p(j|z)] if for all n € N 
and allj eS 

X Pn ili) oe = by. 


iES 
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Exercise 5.28 


Under the assumptions of Proposition 5.8 show that if $ jesTi = 1, then 
H := } jes 746; is the unique invariant measure of our Markov chain. Here 4; 
is the Dirac delta measure at 7. 


Hint p is an invariant measure of a Markov chain én, n € N, if and only if for each 
n EN, the distribution of En equals u, provided the same holds for £o. 


Exercise 5.29 


Show that if 7; = 0 for all j € S, then there is no invariant measure. 


Hint Look closely at the uniqueness part of the solution to Exercise 5.28. 


The following exercise shows that a unique invariant measure may exist, 
even though the condition (5.47) is not satisfied. 


Exercise 5.30 


Find all invariant measures for a Markov chain whose graph is given in Fig- 
ure 5.2. 


Figure 5.2. Transition probabilities of the 
Markov chain in Exercise 5.30 





Hint Find the transition probability matrix P and solve the vector equation Pr = 7 
for m = (71,72), subject to the condition mı + m2 = 1. 


We shall study some general properties of invariant measures. Above we 
have seen examples of Markov chains with a unique invariant measure. In what 
follows we shall investigate the structure of the set of all invariant measures. 


Exercise 5.31 


Show that if u and v are invariant measures and @ € (0, 1], then (1 — 6) + Ov 
is also an invariant measure. 


TT: ‘ A 1 Tes, (Bs a es PE edi C8 ee E S er AE E ESA 
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Exercise 5.32 
Show that if u is an invariant measure of a Markov chain én, n € N with state 
space 5, then supp y C S\T, where T denotes (as usual) the set of all transient 


states. 


Hint If j is a transient state, then pn(j|t) > 0 for all i € S. 


The above result shows that there is a close relationship between invariant 
measures and recurrent states. Below we shall present without proof a couple 
of results on the existence of such measures and their properties. 


Theorem 5.4 


Suppose that én, n € N, is a Markov chain on a state space S = T UC’, where 
T is the set of all transient states and C is a closed irreducible set of recurrent 
states. Then there exists an invariant measure if and only if each element of C 
is positive-recurrent. Moreover, if this is the case, then the invariant measure 
is unique and it is given by = )_, iĝi, where 
hy = — 
Mi 


with m; being the mean recurrence time of the state i, see (5.44). 


Note, that by Exercise 5.32, the unique invariant measure in Theorem 5.4 
is supported by C. 


Remark 5.5 


If C = UG Cj, where each C; is a closed irreducible set of recurrent states, 
then the above result holds, except for the uniqueness part. In fact, if each 
element of some C; is positive-recurrent, then there exists a invariant measure 
ji; supported by C;. Moreover, u; is the unique invariant measure with support 
in Cj. In the special case when each element of C is positive-recurrent, every 
invariant measure u is a convex combination of the invariant measures pj, 


j3€ {1,---,N}. 


Theorem 5.5 


Suppose that n, n € N, is a Markov chain with state space S. Let j € S bea 
recurrent state. 


3 Hence in the decomposition (5.46) the number N of different classes of recurrent 


2D ih iy a eee eek: A 
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1) If j is aperiodic, then 





— 1 
Pnljlj) > —. (5.50) 
ne 
Moreover, for any i € S, 
Fy 
Pn(jlt) > jil 4 (5.51) 
Mj 


where F;;(1) is the probability that the chain will ever visit state 7 if it 
starts at 1, see (5.39), and where m; is the mean recurrence time of state j, 
see (5.44); 


2) If j is a periodic state of period d > 2, then 


Pnaljjj) > (5.52) 


Exercise 5.33 


Suppose that én, n € N, is a Markov chain with state space S. Let 7 E€ S bea 
transient state. Show that for any i € S 


Pn(j|t) > 0. (5.53) 


Hint Use Exercise 5.19. 


Definition 5.11 


A Markov chain €,, n € N, with state space S is called ergodic if each i € S is 
ergodic, i.e. each state z € S is positive, recurrent and aperiodic. 


Exercise 5.34 


Show that if én, n € N, is an ergodic irreducible Markov chain with state 
space S, then pn(jļi) > 7; as n — oo for any j,i € S, where m = mið; is 
the unique invariant measure. 


Hint Use Theorem 5.5. You may assume as a known fact that if j is recurrent and 
i & 7, then Fa) =y 


Exercise 5.35 


Use the last result to investigate whether the random walk on Z has an invariant 


measure. 


Hint Tlea Fvarcica KQ 
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Below we shall see that a converse result to Theorem 5.5 is also true. 


Theorem 5.6 


Suppose that €,, n € N, is an irreducible aperiodic Markov chain with state 
space S. Then én, n € N, is ergodic if and only if it has a unique invariant 
measure. 


Proof 


The ‘if’? part is proved in Exercise 5.34. We shall deal with the ‘only if’ part. 
Suppose that 7 = )/,7;6; is the unique invariant measure of the chain. Then 
nm; > 0 for some j € S. Recall that due to Theorem 5.5 and the Exercise 5.33, 
limn—+oo Pn(j|t) exists for all 7,7 € S. 

Since >, Pn(jlt)a; = tj, by the Fatou lemma (inequality (5.49)) 


> lim pnlili)m 2 lim sup ) Pa (limi = 75. 
? ? 


Hence, there exists an 2 € S such that limp. pn(Jli)mi > 0. Therefore 
limn+oo Pn(j|t) > 0, which in view of Theorem 5.5 implies that m; < œ. 
Thus, 7 is an ergodic state and, since the chain is irreducible, all states are 
ergodic as well. O 


Exercise 5.36 


Prove that if there exists an invariant measure for the Markov chain in Exer- 
cise 5.14, then A’ := D jg <1. Assuming that the converse is also true, 
conclude that the chain is ergodic if and only if A < 1. Show that if such an 
invariant measure exists, then it is unique. 


Hint Suppose that 7 = a 7,6; is an invariant measure. Write down an infinite 
system of linear equations for z,;. If you don’t know how to follow, look at the solution. 


5.4 Long-Time Behaviour of Markov Chains 
with Finite State Space 
As we have seen above, the existence of the 7; plays a very important role 


in the study of invariant measures. In what follows we shall investigate this 
question in the case when the state space S is finite. 
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Theorem 5.7 


Suppose that S is finite and the transition matrix P = [p(j|i)] of a Markov 
chain on S satisfies the condition 


dng E N Je > 0: pr, (Jli) 2€, i,j ES. (5.54) 


Then, the following limit exists for all 1,7 € S and is independent of 1: 


lim pn(jli) = Tj. (5.55) 
n—- oO 
The numbers r; satisfy 
my >0,j€S and X rj=1. (5.56) 
jes 


Conversely, if a sequence of numbers 7, j € S satisfies conditions (5.55)—(5.56), 
then assumption (5.54) is also satisfied. 


Proof 


Denote the matrix P™ = [pno (jli)] by Q = [q(j|z)]. Then the process nk = Exng ; 
k € N, is a Markov chain on S with transition probability matrix Q satisfying 
(5.54) with no equal to 1. Note that pen, (jli) = ae (jli) due to the Chapman- 
Kolmogorov equations. Suppose that the properties (5.55)—-(5.56) hold true 
for Q. In particular, limp_,oo Pkno (Jli) = Tt; exists and is independent of i. We 
claim that they are also true for the original matrix P. Obviously, one only 
needs to check condition (5.55). The Chapman—Kolmogoroy equations (and 
the fact that S is finite) imply that for any r=1,---,ng — 1 


Denotr(Jli) = > Peno (ils)pr (sli) > $ myp,(s|a) 
ses ses 
= Ng X pr(sli) = 
sES 

Therefore, by a simple result in calculus, according to which, if for a sequence 
an, n € N, there exists a natural number no such that for each r € {0,1,---,no— 
1} the limit lim;_,. Gkn +r exists and is r-independent, then the sequence an 
is convergent to the common limit of those subsequences, we infer that (5.55) 
is satisfied. 

In what follows we shall assume that (5.54) holds with no = 1. Let us put 
po(a|t) = Ôji and for j € S 


= 

z 

een, 

Q’. 

War” 
| 


= min Pn(jli) 


M,(j) := max pn(jli) 
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Observe that Mo(j) = 1 and mo(j) = 0 for all j € S. From the Chapman- 
Kolmogorov equations it follows that the sequence M,,(j), n € N, is decreasing, 
while the sequence m,(j), n € N, is increasing. Indeed, since 5°, p(k|i) = 1, 


S > pa (alk)p (RIE) 
kEeS 


min pn(j|k) Š p(kli) 
keS 


min Pn(j|k) = ™Mn(J). 


Pn+i(j|t) 


[V 


I 


Hence, by taking the minimum over all i € S, we arrive at 
Mnti(j) = min Pr+1(j|2) > Mn(j). 


Similarly, 
Pnti(li) = X pa(jlk)plkli) 


keS 

max Pn(J|k) X p(kli) 
kes 

max pn(j|k) = M,(J). 


| 


lA 


Hence, by taking the maximum over all i € S, we obtain 
Mn+1(J) = max pn+1 (jli) < Mn(). 


Since M,,(j) > mn(j), the sequences M,,(j) and Mmn(j) are bounded from below 
and from above, respectively. As a consequence, they both have limits. To show 
that the limits coincide we shall prove that 


Jim, (My(j) = mn(9)) = 0. (5.57) 
For n > 0 we have 


Pai (jli) = X pn(Jls)p(slé) (5.58) 


sES 


$ pn (Ils) [p(sli) — epn (slj)] + € X pn(ls)pnaCsli) 


sES sES 
= 2 PU (jls) [p(s|t) — epn (s|j)] + EP2n (ilj) 


by the Chapman-Kolmogorov equations. The expression in square brackets is 
> 0. Indeed, by assumption (5.54), p(s|i) > € and pn(s|j) < 1. Therefore, 





Raa sacra NETIA AE 
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Pn+i(jlt) 2 min Pals) X [p(slé) — epn (slj)] + Epon (lj) (5-59) 
sES 


= (1 — e)mn(j) + EPp2n (Jlj). 
By taking the minimum over 7 € S, we arrive at 
Mnr+1(J) > (1 — E)mn (j) + EPzn (Jli). (5.60) 


Recycling the above argument, we obtain a similar inequality for the sequence 
M,(j): 


Mn4i(g) < (1 — €)Mn(j) + EP2n (Jlj). (5.61) 
Thus, by subtracting (5.60) from (5.61) we get 
Mn4i(j) — Mn4i(9) < (1 — €) (Mn (3) — Mn (J) . (5.62) 


Hence, by induction 
Manli) — maj) < (1- €)”, nen 


This proves (5.57). Denote by 7; the common limit of Mn (j) and mn (j). Then 
(5.55) follows from (5.57). Indeed, if i,j € S, then 


Mn(j) < Palili) < Mn (j). 
To prove that m; > 0 let us recall that mn(j) is an increasing sequence and 
mi(j) > € by (5.54). We infer that 7; > e. O 
Exercise 5.37 


Show that pn(j|i) > a; at an exponential rate. 


Hint Recall that mn(j) < m; < Mn(j) and use (5.62). 


The above proves the following important result. 


Theorem 5.8 


Suppose that the transition matrix P = [p(j|i)] of a Markov chain €,, n € N, 
satisfies assumption (5.54). Show that there exists a unique invariant measure 
u. Moreover, for some A > 0, anda <1 


lPn(j|t) — mi| < Aa”, ij E Sn EN. (5.63) 
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Proof (of the converse part of Theorem 5.7) 


Put 


E = z min Tj. 
2 j 


Since pn (jli) + 7; for all i,j € S, there is an no € N, such that px(j|t) > e for 
all for k > no and (i,j) € S*. Putting k = no proves that (5.54) is satisfied. 

Let us observe that we have used only two facts: 7; > 0 for all j € S, and 
Pr(jlt) > r; for alli,7 € S. O 


Exercise 5.38 


Investigate the existence and uniqueness of an invariant measure for the Markov 
chain in Proposition 5.1. 


Hint Are the assumptions of Theorem 5.7 satisfied? 


Remark 5.6 


The solution to Exercise 5.38 allows us to find the unique invariant measure by 
direct methods, i.e. by solving the linear equations (5.79)—(5.80). 


Exercise 5.39 -~ 


Find the invariant measure from Exercise 5.38 by calculating the limits (5.55). 


Hint Refer to Solution 5.5. 


Exercise 5.40 


Ian plays a fair game of dice. After the nth roll of a die he writes down the 
maximum outcome én obtained so far. Show that €, is a Markov chain and find 
its transition probabilities. 


Hint €n+1 = max{En, Xn4+1}, where Xx is the outcome of the kth roll. 


Exercise 5.41 


Analyse the Markov chain described in Exercise 5.40, but with fair die replaced 


by a fair pyramid. 


Hint A pyramid has four faces only. 
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Exercise 5.42 


Suppose that a > 1 is a natural number. Consider a random walk on S = 
{0,1,:--,a} with absorbing barriers at 0 and a, and with probability p of 
moving to the right and probability q = 1 — p of moving to the left from any 
of the states 1,...,a — 1. Hence, our random walk is a Markov chain with 
transition probabilities 


if 1<i<a—-1,j=i41, 
if 1<i<a-l1,j=i-1l, 


i Vey = 0: Or tS Sa, 
otherwise. 


p(j\t) = 


Or QS 


Find, (a) all invariant measures (there may be just one), (b) the probability of 
hitting the right-hand barrier prior to hitting the left-hand one. 


Hint For (a) recall Exercise 5.30 and for (b) Exercise 5.12. 


5.5 Solutions 


Solution 5.1 ; 


First we give a direct solution. With A, and B, being the events that the phone 
is free or busy in the nth minute, we have yn = P(B,). The total probability 
formula then yields, for n € N, 


P(Bn+1) = P(Bn41|An)P(An) + P(Bn4i|Bn)P(Bn) 
1.e. 
Yn+1ı =P + (1 -—p-aQ)yn. (5.64) 


As before, assuming for the time being that y = limn Yn exists, we find that 
y=pt+(l—p-—q)y, and soy = naar In particular, AT =pt+(l-p- aa 
Subtracting the last equality from (5.64), we see that {yn — a] is a geometric 
sequence, so that Yn — a = (vo — 2) (1—p-q)". Since yo = 1, some simple 
algebra leads to the formula 


Yn n eaa (1—p-q)", nEenN. 


= p+4q p+q 
The last formula can be used to prove that the lim, y, exists and is equal to 


Lf. i 
p+q 
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Another approach is to use the results of Example 5.1. Since zg = 1 — yo, 
by (5.8) we have 


q q n 
AEE E ee 1-w- -4 ) 0 -p-0) 
( ° p+q 


Yn 
aei te q ( 
ptrq prq 
which agrees with the first method of solution. 

Solution 5.2 

We have to show that: >) jes PU li) = 1 for every i € S. We have 


>> pili) = So PG = lh = å) 


jes JES 
= P(Ujes{& = Sho = i) = P(& € Sif = i) 
= P(Q =1i)=1. 


Solution 5.3 


Suppose that A = [ajiljics and B = [bji]jics are two stochastic matrices. If 
C = BA, then cji = $, bjkaki. Hence, for any i € S 


j 


= X | dyn | aes = ð ane = 1, 
k j 


k 


where the last two equalities hold because B and A are stochastic matrices. 
We have used the well-known fact that 


> aj = XS X ay, 5.65) 
t J J t 

for any non-negative double sequence (a;;)?4_, (see, for example, Rudin’s book 
cited in the hint to Exercise 5.18). The above argument implies that P? is a 
stochastic matrix. The desired result follows by induction. 

To prove that P” is a double stochastic matrix whenever P is, it is enough 
to observe that AB is a double stochastic matrix if A and B are. The latter 
follows because (AB)’ = Bt A’. 





of 
ca SSA 
an que ARH TE IRS RESET AT NTE IRI ETN - nee 
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Solution 5.4 
Some simple algebra gives 
p2 — pe ak ae 
pil—p)+(1-—q)p pqt+(l-q)(1-4@) 
M a ace 2q — pq — q° 
2p- pq=-p° 1+- 2q+pq 
Solution 5.5 


Put zn = P(E, = Olf = 0) and yn = P(En = 1ļ|o = 1). We have calculated 
the formulae for zn and yn in Example 5.1 and Exercise 5.1. Since also 


p p 

fy = a a 
q+p q+p 
q q 

l- yn = —— - — (1 -p-q)”, 
q+p Fee 


we arrive at the following formula for the n-step transition matrix: 


P, = (5.66) 





ae ee eee ey Gee ee S EE ary ees e 
ap + a0) p — q)” q+p epl p A 


EES eG ee 9 SS ae gee Ca fy eee eer 
q+p pll p — q)” ate gr) p — 4) 


Solution 5.6 


Simplifying, we have 


Aedes e e 
Py = | age * aE p-4q) q+p ap Ul p-4q) | 


P _ = = 2 = _ 2 
pi apr oe) Ga a 
eeerned sce 
—(q-2+p)p q?—2q¢+qptl1 |’ 


which, in view of the formula in Exercise 5.4, proves that P> = P?. 

We shall use induction to prove that P, = P” for all n € N. We already 
know that the assertion true for n = 1 (and also for n = 2). Suppose that 
Pa = P”. Then some simple, but tedious algebra gives 


Phiri p a 

Ty q | aoe ip aP- a" 
n ais z= z n —_ — n 
p sai q+p zp ll p-4) ate + apt p-4q) 

— p — TE e T a +l 

= apt ash i A p 7a A ye 

2 n — n — n 

ae gee atp t gpl —P— 4) 





| 


Pea 
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Solution 5.7 


Recall that P+: = P"P*. Since pn+4(j|i) are the entries of the matrix Pr+ x = 
prtk = P” P* we obtain (5.18) directly from the definition of the product of 


two matrices. 

Solution 5.8 

Since the n; are independent, En and n41 are also independent. Therefore, 
P (Enti = s|fo = $0,°'',€n = Sn) 
a P (En + Mn41 = s|éo = een = Sn) 


P (41 = 8 — 8n|€0 = 80, En = Sn) 


P (nar SS Sx): 


Similarly, 
P (€n41 = 8[€n = Sn) = P (En + Mnti = 8|En = Sn) 


P (n+ TES Sn|En = S54) 
P (Nn41 = 8 — Sn). 


| 


Solution 5.9 


Step 1. For n = 1 the right-hand side of (5.19) is equal to 0 unless |j — i| < 1 
and 1+ j — i is even. This is only possible when 7 = i+ 1 or j = ¿i — 1. In 
the former case the right-hand side equals p, and in the latter it equals q. ‘This 
proves (5.19) for n = 1. 
~ Step 2. Suppose that (5.19) is true for some n. We will use the following 
version of the total probability formula. If H; € F, P(H; N H;) = 0 for i £ j, 
and P(\), Hi) = 1, then 


P(A|C) = X` P(AIC A Hi) P(HiIC). (5.67) 


Then the Markov property and (5.67) imply that 


P(Enti = J/€o = 7) 

PlEnti = j/fo = t, En = j —1)P(En = j — 110 = 2) 
+P(n41 = jlfo = i, En = j + 1)P(En = j + 110 = 4) 
PEn+i = Jln = J — 1)P(En = j — 1]€o = 2) 
+P(En4i = ln = j +1) P(En = j + 1h = i) 


n n+j—l—i n-~j+i+i n n+j+l-i n—j—1+i 
= p EE p 2 q 2 +q aidjili=i p 2 q 2 
2 2 


l 


II 
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| 


n n n+l+j-i n+l-j+i 
(aii) at (tia) P 4 q 
2 2 


n+l n+14j-i n+1—j+i 
Pani a q : ° 
2 
Solution 5.10 
Denote the right-hand side of (5.24) by h(z). It follows by induction that 


| 1 
hE) (x) = Cn (1 -— 7 a ama |z| < A 


On the other hand, h is analytic and 
h(x) = 5 + 1) (0) 2" |r| < 
k=0 k' | : 


Solution 5.11 


We shall translate the problem into the Markov chain language. Denote by S 
the set of all natural numbers N = {0,1,2,---}. Let En denote the number of 
males in the nth year (or generation), where the present year is called year 0. If 
En = 1, i.e. there are exactly i males in year n, then the probability that there 
will be 7 males in the next year is given by 


P (Ent = dlén = i) = P(X1 ++ Xi = j), (5.68) 


where (X;,)?2, is a sequence of independent identically distributed random 
variables with common distribution 


P(X, =M)=pm, MEN 
Hence én, n > 0 is a Markov chain on S with transition probabilities 
p(j|t) = P(X +--+ Xi = j). (5.69) 


Notice that p(0|0) = 1, ie. if &n = 0, then Em = 0 for all m > n. Dying out 
means that eventually €, = 0, starting from some n € N. Once this happens, 
En will stay at 0 forever. 


Solution 5.12 


We shall only deal with part 2), as in part 1) there is nothing to show. Suppose 
that = i. Then & = X1 + ---X;, where X; are independent identically 
distributed Poisson random variables with parameter A. Since the sum of such 
random variables has the Poisson distribution with parameter iA, (5.27) follows 
readily. 
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Solution 5.13 


Let En denote the number of supporters at the end of day n. Then En+1 — En is 
equal to 1 or 0 and 


Pq, fj=i+l, 
pult)=4{ 1-pq, ifj=i, (5.70) 
0, otherwise. 


Thus €, is a Markov chain on a state space S = N with transition probabilities 
p(jli) given by (5.70). 


Solution 5.14 


We shall use the notation introduced in the hint. Observe that En+1 — En — Zn 
equals —1 or 0. The latter case occurs with probability p, i.e. when the car 
served at the beginning of the nth time interval was finished by the end of the 
nth time interval. The former case occurs with probability 1 — p. Therefore 


\i-itl Ja 


P n = 9 n = i e =A 1 Ca O —A 
for i > 1,7 > 1— 1. On the other hand, if j > 0, then 
A 
P(En+1 = JlẹEn = 0) = ri 
Thus, €, is a Markov chain with transition probabilities 
plil) = 4 Gigi if ¿> 1,j>2i-l, (5.71) 
0, otherwise, 
where qk = Ae and 
! Pak; if k= 0, 
= 5.72 
- | (1— p)qk-1 + Pak, if k>1. =m 


The transition probability matrix of our chain takes the form 


[o a 0 0 


q g Q 0 
@ o ú Q O0 
Ps | : : / 


Q d 
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Solution 5.15 


If px (jli) > 0, then P(E, = j for somen > O|& = i) > pe(jlé) > 0. If pa (jli) = 0 
for all k > 1, then 


P(En =Jj for some n 2 Olo = 1) < > Pia = 9|£0 = 1) a X Pn(Jlt) = 0. 
n=l n=l 


Solution 5.16 


Since P(o = i|/€) = i) = 1, it follows that i + i, which proves 1). Assertion 2) is 
obvious. To prove 3) we proceed as follows. From the solution to Exercise 5.15 
we can find n,m > 1 such that pn(jli) > 0 and pm(k|j) > 0. Hence, the 
Chapman-Kolmogorov equations yield 


Pm+n(kli) = X Pm(kls)pn(sli) > Pm(kli)pn(jli) > 0. 
sES 


Solution 5.17 


Since |pn(jli)| < 1 and |fn(j|t)| < 1 for all n € N, the radii of convergence of 
both power series are > 1. To prove the equalities (5.40)-(5.41) we shall show 
that for n > 1 and any i,j € S, 


Pnlili) = X fe CGli ph- (Jl). (5.73) 
k=1 Š 


By total probability formula and the Markov property 
Pnrljli) = P(n = jlo =?) 
= $ Plen =A = KG FALSE Sk Mh = i) 
k=1 
=) Pe =7,4A44,1<1< k-1l = i) 
k=1 
xP(En = Ibe =F, Z j,1 <L< k- 1) 


= Sfx (lt) P En = j|€ = j) 
k=1 

= Y flipa- lli). 
k=1 


Solution 5.18 
Since 0 < pn(jli) < 1 and 0 < falli) < 1, the result follows readily from 


Ald 19. 12 
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Solution 5.19 


Only the case of a recurrent state needs to be studied. Suppose that j is recur- 
rent. Then $7; falili) = k Hence, by Exercise 5.18, Fj;(z) Z l asz Z1. 
Thus P;;(x£) = (1 — Fj;(z))~' —> 00 as z 1, and so, ier by Exercise 5.18, 
yo 1 Pn(Jlj) = œ. Conversely, suppose that $>; pn(j|j) = oo. Then, by 
Exercise 5.18, P;;(z) = œo asx 71. Thus, F;;(x) = 1 — (Pj;(z))~* > 1 as 
a 71. Hence, Xp; fr(jlj) = 1, which proves that j is recurrent. 

To prove (5.43) we use (5.73) to get 


Ym WO 
==0 


œo n-l 


SoS fn—e (Sli) pe GIS) 


n=0 k=0 


= 3 fn(Jli)pe (313) 


k=0 m=1 


= 5 pe (ili) 3 fn (Jt) 
k=0 m=1 

< So pe (sls). 
k=0 


This implies (5.43) when 7 is transient. 


Solution 5.20 


We shall show that the state 0 is recurrent. The other case can be treated in 
a similar way. From Exercise 5.8 we have p,(0|0) = -f + -2-(1—p—q)”. 
Thus p,(0|0) > —- sty > 9, and so YO o Pn(0|0) = co, which proves that 0 is a 


recurrent state. 


Solution 5.21 


Suppose each j € S is transient. Then by (5.43) 3°, pn(jlt) < for all 
i € S. Let us fix i € S. Then we would have } jes 07-1 Pn(Ht) < œ, 
since S is finite. However, this is — because )/ ics Pag) = 


nai jes Pn lt) = Erai 1 = 
Solution 5.22 


Let us begin with a brief remark concerning the last part of the problem. Since 
the random walk is ‘space homogenous’, i.e. p(j|t) = p(j—il0), it should be quite 
obvious, at least intuitively, that either all states are positive-recurrent.or all 
states are null-recurrent. One can prove this rigorously without any particular 
difficulty. First, observe (and prove by induction) that pr(j|t) = pn(j — i|0). 







5. Markov Chains 127 


Second, observe that the same holds for fn, i.e. fn(jli) = fn(j — il0). Hence, in 
particular, m; = mo. 
To show that 0 is null-recurrent let us recall some useful tools: 


OO 
Poo(z) = ) > Pn(0|0)e", -1< 2 <1, 
n=0 


Foo(z) = X fa(0|0)a", -1<2< 1. 
n=l 


2k\ 1 
k J 4k? 


Poo(x) = 3 Ce) ey = (1-27). 


k=0 


Since, see Exercise 5.9, 


por (00) 


Then, using (5.41), we infer that Foo(x) = 1—(1—27)!/?. Since F(x) A œ as 
zr 7 l and Foo(z) = } pı nfn(0|0)z" by using Abel’s lemma (compare with 
Exercise 5.18), we infer that $7; nf,(0|0) = oo. This shows that mo = oo 
and thus 0 is null-recurrent. 


Solution 5.23 


We know that 0 is a recurrent state. From definition 
fr(0|0) = p(O|1)p(1|1)"~*p(1 0) = pa(1 — q)”~?. 


Since |1 — q| < 1, we infer that X` >, nfn(0|0) = $> pq(1 — g)"~? < ow. 
Hence mg < co and 0 is positive-recurrent. The same proof works for state 1. 


Solution 5.24 


Consider a Markov chain on S = {1, 2,---,9} with transition probabilities given 
by the graph in Figure 5.3. Then, obviously, p4(1|1) = 1/2 and pe(1|1) = 1/2, 


Figure 5.3. Transition probabil- 
ities of the Markov chain in Exer- 
cise 5.24 





but p,(1]1) =O if k <6 and k ¢ {4,6}. Hence d(1) = 2, but pa(1|1) = 0. 
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Solution 5.25 


We begin with finding P) = P? in an algebraic way, i.e. by multiplying the 
matrix P by itself. We have 


2 0 1/2 0 1/2 | _ 
r=P=|] ajli i ]=| 

Alternatively, P> can be found by observing that the only way one can get from 
1 to 1 in two steps is to move from 1 to 2 (with probability 1) and then from 2 
to 1 (with probability 1/2). Hence, the probability po(1|1) of going from 1 to 1 
in two steps equals 1/2. Analogously, we calculate p2(1|2) by observing that in 
order to move from 1 to 2 in two steps one needs first to move from 1 to 2 (with 
probability 1) and then stay at 2 (with probability 1/2). Hence, po(1|2) = 1/2. 
The remaining two elements of the matrix P can be found by repeating the 
above argument, or, simply by adding the rows so that they equal 1. In the 
latter method we use the fact that P> is a stochastic matrix, see Exercise 5.3. 
The graph representing P> is shown in Figure 5.4. 


bole bole 
Afw ajem 


1/2 


1/2 3/4 


Figure 5.4. Two-step transition 


1/4 
í probabilities in Exercise 5.25 


Using any of the methods presented above, we obtain 


? 


Therefore, pı(1|1) = 0, p2(1[1) = 1/2 and p3(1|1) = 1/4. Hence d(1) = 1 
(although p; (1/1) = 0). Since pı (2/2) = 1/2 > 0, it follows that d(2) = 1. 


œen ow 


n= 


Ca heed Lae 


Solution 5.26 


. Suppose that o = i. Denote by 7, the minimum positive time when the chain 
\enters state k, i.e. 
T, =min{n>1:€, =k}. 
Then, P{7; < Tti} =: € > Oifi > j andi Æ j. If j » i, then it would 
be impossible to return to i with probability at least £ > 0. But this cannot 
happen as 7 is a recurrent state. Indeed, 


L = P (7; < o0) aP < Ool < 7) P (7; < Ti) 
PPU orma AT): 
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The second term on the right-hand side is < 1 — e€ < 1, while the first factor in 
the first term is equal to 0 (since i » j). This is a contradiction. The second 
part is obvious. 


Solution 5.27 


As observed in the hint, we only need to show that (5.45) holds when p(j|i) = 0 
for i € C and 7 E€ S\C. We begin with the following simple observation. If 
(§2,F, P) is a probability space and A, E F, n EN, then P(U, An) = 0 if and 
only if P(A,) = 0 for each n € N. Hence (5.45) holds if and only if for each 
k>n 
P (£x E€ S \ Clên E€ C) =0. (5.74) 

In fact, the above holds if and only if it holds for k = n + 1. Indeed, suppose 
that for each n € N 

P (n41 E€ S VClE, E C) = 0. (5.75) 
Let us take n € N. We shall prove by induction on k > n that (5.74) holds. This 
is so for k = n and k = n + 1. Suppose that (5.74) holds for some k > n + 1. 
We shall verify that it holds for k + 1. By the total probability formula (5.67) 
and the Markov property (5.10) | 


P (€r41 E S\ ClEn E C) 
= P (k41 E€ S \ Cl|ên € C, ék E€ C) P(Ek € Clin E C) 
+ P (k41 E S \ C lEn E C, Ek E S\C) PlEk € S\ Cl&n E C) 
= P (E11 € S \ Cl& E C) P(& E€ Cl& E C) 
+ P (k41 €ES\Cl& € S\C)P(êk E€ S \ Cl& € C). 
By the induction hypothesis P(€ € S\C|En € C) = 0 and by (5.75) (applied to 
k rather than n) P (€&k+1 € S \ Cl& € C) = 0. Thus, P (&41 € S\ Cl&, € C) = 
0, which proves (5.74). 
The time-homogeneity of the chain implies that (5.75) is equivalent to (5.74) 


for n = 0. Since P is a countably additive measure and S is a countable set, 
the latter holds if and only if p(j|t) = 0 for i € C and j € S \C. 


Solution 5.28 


Property 2) in Proposition 5.8 implies that u is an invariant measure. Therefore 
it remains to prove uniqueness. Suppose that v = Da qj0; is an invariant 
measure. It is sufficient then to show that 7; = q; for all j € S. 

Since 0 < pr(jlt)qgi < qi for all i,j € S and $°,q; = 1 < oo, Lebesgue’s 
dominated convergence theorem yields 


OQ 
qj = S| pn(ili)ai a X Tjqi = fj. 


130 Basic Stochastic Processes 


It follows that u = v. 


Solution 5.29 


We shall argue as in the uniqueness part of Solution 5.28. If v = }7°, qj4; is 
an invariant measure, then by Lebesgue’s dominated convergence theorem 


OO 
gj = > palslia > >> rjq: = 0. 
i=1 i 
Hence v = 0, which contradicts the assumption that v is a probability measure. 


Solution 5.30 
Obviously, P = | i ; | . Therefore equation Pr = m becomes 
Ti = N2, 
T2? = Tl. 
The only solution of this system subject to the condition mı + m2 = 1 is m7, = 
No = 1/2. 
Solution 5.31 
Put u = Ì jcs Mid; and v = 0-5 viði. Then, for any j E€ S and n € N 
X palli) (1 — us + 0%) = (1-8) $ pr(ili)ui + OY” pn(jli)vi 


iES | ies iES 
(1 — @) py; + Ov; = [(1 — 0)u + 0v); . 


Solution 5.32 
Put u = ics Hid;. Then, for any j € T 


uj = X pn(jli)ui > 0 
iES 
by the Lebesgue dominated convergence theorem. Indeed, by Exercise 5.19 
Pn(jli) — O for all 2 € S, and py(yt) i < pi, where È` ui < œœ. 
Solution 5.33 


Since j is transient in viewOf (5.43), from Exercise 5.19 we readily get (5.53). 






Solution 5.34 
By Theorem 5.5 pp(jli) > a for all i,j € S. Put 7; = =e j E€ S. We need to 
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1) m = )/,7;6; is an invariant measure of the chain €,, n € N; 
2) v is the unique invariant measure. 


Part 1) will follow from Proposition 5.8 as soon as we can show that m; > 0 for 
at least one 7 € S. But if 7 € S, then j is positive-recurrent and thus m; < oo. 
Part 2) follows from Exercise 5.28. 


Solution 5.35 


Let us fix i,j € Z. First suppose that j — i = 2a € 2Z. Then poz41(7|2) = 0 for 
all k € N, and also pox(j|t) = 0 if k < |a|. Moreover, if k > ja], then 


inf ak N keakea n (L) (2k \ ig 
p2k(jli) = FAL q = (2) fra Pq. 
Since (25) < (7°), it follows that 
ae * (2k 
por(jli) < (2) E Jae +0 


by Proposition 5.4. We have therefore proven that 7; is well defined and that 
nm; = 0 for all 7 € Z. Hence, we infer that no invariant measure exists. 


Solution 5.36 


Suppose that 7 = ee, 7;0; iS an invariant measure of our Markov chain. 
Then $; p(jli)ni = mj for all j e S = N. Using the exact form of the transition 
probability matrix in the solution to Exercise 5.14, we see that the sequence 
of non-negative numbers (7;)329 solves the following infinite system of linear 
equations: 


GoTo + gori = To 
To +q nmi + gar = J 
ae a ; (5.76) 
Q2no + qanı + 9172 + QoT3 = TM 
l.e. 
k+1 
Qk + Š dkti-jTj = Tk, keN. 
j=l 


Multiplying the kth equation in (5.76) by zë, k > 0, and summing all of them 
up we obtain 


ToQ(x) + 


J=l 
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where for |x| < 1, 


Since ) 7<; 7j27 = II (x) — 70, we see that I, Q and G satisfy the functional 

equation 

G(x) — cQ(z) 
G(x) -a ` 

Since all the coefficients m; in the power series defining I are non-negative, 

Abel’s lemma implies that 


IT(x) = To iz) < 1. (5.78) 


2 i= m IT(x). 
j=0 


It is possible to use (5.78) to calculate this limit. However, there is still some 
work be done. We have to use l’Hospital’s rule. Since G(x) — x —> 0, G(x) — 
tQ(r) > 0, G'(z) —1 > A! — 1 and (G(x) — zQ(z))' => X' — X— 1 (all limits 
are for z 1), we obtain (recall that A’ = lim, 71 G’(z)) 
1) if #1, then H(z) > To HHA; 

2) if A‘ = 1, then H(z) > oo. 

In case 1) lA A > 1, since mo < 1. It follows that zA > 0, and so X < 1. 
Moreover, in this case 


1-A 
To = ——. 
° 1=N+A 
The above argument shows that if there exists an invariant measure, then it is 


unique. 

Now suppose that an invariant measure exists (and then it is unique). Since 
our chain is irreducible and aperiodic (check this!), it follows from Theorem 5.6 
that the Markov chain €, is ergodic. Therefore, by Theorem 5.5, for any 7 € N 


Pr(JlJ) > Tj, asn — ov, 
Since F;;(1) = 1, it also follows that 


Pn(j|t) = Tj, as n —> ow. 
/ 


/ 
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Solution 5.37 


We begin with the case no = 1. Then M,(j) — mn(j) < (1 — £)” from (5.62). 
It follows that 


Mn(j) < Nj < M,(J), 
Mn(J) < Pn(slt) < MnC), 


and we infer that |p,(j|t) — 7;| < (1 — €)”, n € N. Suppose that no > 2. Then 
for any r = 1,--:,no — 1 


XC Pino (jls)pr (sli) — È tyr (sli) 


Pinar (ili) — Nj | 





sES ses 

< X` [Peno (jls) — Tilpr(sli) 
sES 

< max |Pkno (jls) — T|X Pr(sli) < (1 - £)”. 
“<2 sES 


Solution 5.38 

1 — 
The transition probability matrix P takes the form | . p a | Since 
all four numbers p,q,1 — p,1—q are strictly positive, the assumption (5.54) 
is satisfied, and so the limits 7; = lim, pn(jli) exist and are i-independent. 
Hence, the unique invariant measure of the corresponding Markov chain is 
equal to 7969 + 76. (Recall that S = {0,1} in this example.) We need to find 
the values 7;. One way of finding them is to use the definition. As in Hint 5.28, 
the vector 7 = (To, mı) solves the following linear equation in matrix form: 


ae fetal S| (679) 
motm = 1. (5.80) 


Some elementary algebra allows us to find the unique solution to the above 
problem: 


q P 
meae E (5.81 
p+q ptq ) 
Hence, reat + ete is the unique invariant measure of the Markov chain. 


Solution 5.39 
From (5.66) we infer that 


fi 4 —F (1-p—g)" + +, 


Pn(0|0) = —— 
n(0 q+p qtp qt+p 
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p p p 


n(110 == = (1-p-q > ——, 
P eae E 
q q a q 
pr(0j1) = — - — (1 -p- q)" > —, 
| q+p q+p q+ p 
p q : p 
a a ee et ee 
(| q+p qtp q+p 


Hence 7 = ae and mı = oro This is in agreement with the previous solution. 


Solution 5.40 


Suppose that E» = i. Then the value of £,,; depends entirely on the outcome of 
the next roll of a die, say Xn+1, and the value of En. Since it depends on the past 
only through the value of €,, intuitively we can see that we are dealing with a 
Markov chain. To define €, in a precise way consider a sequence of independent 
identically distributed random variables Xn, n = 1,2,3,--- such that P(X, = 
i) = 1/6 for alli = 1,2,---,6. Then, putting €,41; = max{€,, Xn+1 }, we can see 
immediately that P(En+1 = tn4ilo = t0,°--.€n = tn) = PlEngi = tngil€n = 
in) and 


Pbnt1 = If =t) = PlEn4i = Ibn = i, Xngi < 1)P(Xn4i <1) 
+P(En4i = J|En = i, Xn41 > 1)P(Xn41 > å) 
o sss 





. 6—i EA . e 
=$ 1 ijai ett a izn 
0 ifj<i, HJ at 
1 iff >4, 
Sa ijen 
0 iff <i. 
Thus, 
- if j >4, 
pll) =4 5 Ifj=i, 
0 iff <i. 
It follows that 
1 000 00 
2 
fees: 
= = > 0 00 
P=]; $ $ $, 
ggg lO 
1 i 1 1 5 g 
6 6 6 6 6 
1l 1l 1 i 1] 
6 6 6 6 6 


Solution 5.41 


The sranh af the chain is given in Figure 5.5. The transition matrix is 


R E O OEO cc css EERTE EOE ENEE = 
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Figure 5.5. Transition probabil- 
ities of the Markov chain in Exer- 
cise 5.41 
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Either from the graph or from the matrix we can see that i > j if and only if 
i < j. The state i = 1 is transient. Indeed, p(1|1) = 1/4 and pn(1|1) = (1/4)” 
by induction. Therefore, >), pn(1|1) < œœ and 1 is transient by Exercise 5.19. 
The same argument shows that the states 2 and 3 are also transient. On the 
other hand, p(4|4) = 1, and so 4 is a positive-recurrent state. (As we know, 
that there should be at least one positive-recurrent state.) 

We shall find invariant measures by solving the system of four linear equa- 
tions Pr = n for mn = (n1, t2, n3, m4), Subject to the condition 7, + m2 + 
m3 + m4 = 1. Some elementary linear algebra shows that the only solution is 
Tı = T3 = 73 = 0, m4 = 1. Thus, the unique invariant measure is 7 = d4. 

The invariant measure can also be found by invoking Theorem 5.4. In our 
case C = {4}, and so there is exactly one class of recurrent sets. Moreover, as 
we have seen before, 4 is positive-recurrent. Therefore, there exists a unique 
invariant measure. Since its support is contained in C, we infer that m = 64. 


Solution 5.42 


The graph representing the Markov chain for a = 4 is presented in Figure 5.6. 
Obviously, i + j for i € S \ {0,a} =: S and j € S. Moreover, 0 — j if and 
only if j = 0 and a — j if and only if 7 = a. Since p(0|0) = p(ala) = 1, 
both states 0 and a are positive-recurrent. All other states are transient. For 


(0) 
if 7 € S were recurrent, i would be intercommunicating with 0 because i —> 0, 
hv Fixercise 5.26. This is impossible. Therefore. by Remark 5.5 there exist an 
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Figure 5.6. Transition probabil- 
ities of the Markov chain in Exer- 
cise 5.42 





infinite number of invariant measures: u = (1 — @)d9 + ôa, where 0 € [0,1]. 
Indeed, ĝa is the only probability measure with support in the singleton {a}. 
It is possible to verify this with bare hands. Next, let ¢(1) denote the probabil- 
ity that the investigated Markov chain €, hits the right-hand barrier prior to 
hitting the left-hand one. Once €, hits the left barrier it will never leave it, so 
(i) is actually the probability that En hits a. Put 


A= {dneN: & =a}. 


Then, by the total probability formula and the Markov property of n, we have 
forl<i<a-l1 


P(Al& = 1) = X` P(AlG =i, & = J) P(E = jlo = i) 


ott) = 
j=0 
= >) P(AI& = j)P(& = jl& = i) 
7=0 
= >) 6) P(& = jléo = i) = poli + 1) + go(i - 1). 
j=0 
Obviously, 
o(0) = 0, 
g(a) = 1 
Therefore, the sequence (¢(i));—o satisfies 
(0) = 0, g(a) = 1. 


Since p+ q = 1, equations (5.82) can be rewritten as follows: 


pll + 1) — ¢(i)] = q [ġli) -d@-1)], 1<i<a-1. 
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Hence, | 
| weli a 
[p(t + 1) - di) = (2) 


where zx = $(1) — $(0) is to be determined. Using the boundary condition 
(a) = 1, we can easily find that 


Y- (li +1) - 6) 
i 


Sg as 
Tis 
l 


1 = ¢(a) 


Here we assume that p # q. The case p = q = 1/2 can be treated in a similar 
way. In fact the latter is easier. It follows that 





= —] 
eee) ees 
and therefore 
tl (2)? Bere | 
oli) = X (b(k +1) — o(k)) = Be 
k=0 (3) l 
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Stochastic Processes in Continuous Time 


6.1 General Notions 


The following definitions are straightforward extensions of those introduced 
earlier for sequences of random variables, the underlying idea being that of a 
family of random variables depending on time. 


Definition 6.1 


A stochastic process is a family of random variables £(t) parametrized by t € T, 
where T C R. When T = {1,2,...}, we shall say that €(t) is a stochastic process 
in discrete time (i.e. a sequence of random variables). When T is an interval 
in R (typically T = [0,00)), we shall say that €(t) is a stochastic process in 
continuous time. | 

For every w € §2 the function 


T3th &(t,w) 


is called a path (or sample path) of €(t). 


Definition 6.2 


A family F, of o-fields on N parametrized by t € T, where T C R, is called a 
filtrationNif 






Fx CS CF 


far anv e t E\T anch that e< t 
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Definition 6.3 


A stochastic process £(t) parametrized by t € T is called a martingale (sub- 
martingale, supermartingale) with respect to a filtration F; if 


1) €(t) is integrable for each t € T; 


2) E(t) is Fy-measurable for each t € T (in which case we say that €(t) is 
adapted to F); 


3) €(s) = E (E(t)|F,) (respectively, < or >) for every s,t € T such that s < t. 


In earlier chapters we have seen various stochastic processes, in particular, 
nartingales in discrete time such as the symmetric random walk, for example. 
n what follows we shall study in some detail two processes in continuous time, 
1amely, the Poisson process and Brownian motion. 


3.2 Poisson Process 


3.2.1 Exponential Distribution and Lack of Memory 


definition 6.4 


Ne say that a random variable n has the exponential distribution of rate A > 0 
f 
Pinst=e~ 


or all t > 0. 


For example, the emissions of particles by a sample of radioactive material 
or calls made at a telephone exchange) occur at random times. The probability 
hat no particle is emitted (no call is made) up to time t is known to decay 
xponentially as t increases. t is to say, the time 7 of the first emission has 
he exponential distribution, P {n >t 


-xercise 6.1 
Vhat is the distribution function of a random variable n with Exponenual 


istribution? Does it have a density? If so, find the density. 


lint What is the probability that 7 > 0? What is the probability that 7 > t for any 
iven t < 0? Can you express the distribution function in terms of P {n > t}? Is the 
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Exercise 6.2 
Compute the expectation and variance of a random variable having the expo- 


nential distribution. 


Hint Use the density found in Exercise 6.1. 


Exercise 6.3 


Show that a random variable 7 with exponential distribution satisfies 
P{n>t+s}=P{n>t}P{n> s} (6.1) 
for any s,t > 0. 


Hint When the probabilities are replaced by exponents, the equality should become 
obvious. 


Exercise 6.4 


Show that the equality in Exercise 6.3 is equivalent to 
P{n>t+s|n >s}=P{n> t} (6.2) 
for any s,t > 0. 


Hint Recall how to compute conditional probability. Observe that 7 > s + t implies 
nN >s. 


The equality (6.2) (or, equivalently, (6.1)) is known as the lack of memory 
property. The odds that no particle will be emitted (no call will be made) in 
the next time interval of length t are not affected by the length of time s it has 
already taken to wait, given that no emission (no call) has occurred yet. 


Exercise 6.5 


Show that the exponential distribution is the only probability distribution sat- 
isfying the lack of memory property. 


Hint The lack of memory property means that g(t) = P {n > t} satisfies the func- 
tional equation 

g(t + s) = g(t)g(s) 
for any s,t > 0. Find all non-negative non-increasing solutions of this functional 
equation. 
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6.2.2 Construction of the Poisson Process 


Let 71,772,..- be a sequence of independent random variables, each having the 
same exponential distribution of rate A. For example, the times between the 
emissions of radioactive particles (or between calls made at a telephone ex- 
change) have this property. We put 


En =m +::' +, 


which can be thought of as the time of the nth emission (the nth call). We also 
put ĉo = 0 for convenience. The number of emissions (calls) up to time t > 0 is 
an n such that £,41 > t > én. In other words, the number of emissions (calls) 
up to time t > 0 is equal to max {n : t > én}. 


Definition 6.5 


We say that N(t), where t > 0, is a Poisson process if 


N(t) = max {n : t > én}. 


Thus, N (t) can be regarded as the number of particles emitted (calls made) 
up to time t. It is an example of a stochastic process in continuous time. A 
typical path of N (t) is shown in Figure 6.1. It begins at the origin, N (0) = 0 
(no particles emitted at time 0), and is right-continuous, non-decreasing and 
piecewise constant with jumps of size 1 at the times £n. 
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Figure 6.1. A typical path of N(t) and the jump times £n 
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What is the distribution of N(t)? To answer this question we need to recall 
the definition of the Poisson distribution. 


Definition 6.6 


A random variable v has the Poisson distribution with parameter a > 0 if 
n 


aa 
P{v=n}s=e i 


for any n =0,1,2,.... 


The probabilities P {v = n} for various values of a are shown in Figure 6.2. 





0123 n 012345 0123456789 nN 
a=1/3 a=] a=3 





Figure 6.2. Poisson distribution with parameter a 


Proposition 6.1 


N(t) has the Poisson distribution with parameter At, 


aa (At)” 
P{N(t) =n} =e a (er 


Proof 


First of all, observe that 


{N(t) <n} = {& > t}. 
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It suffices to compute the probability of this event for any n because 


P{N(t)=n} = P{N(t) <n+1}- P{N(t) <n} 
= P {n41 > t} — P {En > t}. (6.3) 
We shall prove by induction on n that 
Plasie H i | (6.4) 


Forn = 1 
P{& >t}=P{m>t}=e™ 


Next, suppose that (6.4) holds for some n. Then, expressing £,41 as the sum 
of the independent random variables €, and 7n+i1, we compute 


P {Engi > t} = P {En +41 > t} 
P {nn+1 > t} +P {En > t — Nnt > Mmaa > 0} 


t 
e +j P {En >t— 8} fm41(8)ds 
ew ht 4 + fe —X(t—s) y oe Ae — 8) he >8ds 
k+1 
= ent 4 ren À avi (t — 


| 





where f,,,,,(8) is the density of nn+1. By induction, (6.4) holds for any n. Now 
apply (6.3) to complete the proof. O 


Exercise 6.6 


What is the expectation of N(t)? 


Hint What are the possible values of N(t)? What are the corresponding probabilities? 
Can you compute the expectation from these? To simplify the result use the Taylor 
expansion of e”. 


Exercise 6.7 
Compute P {N(s) = 1, N(t) = 2} for any O<s <t. 


Hint Express {N(s) = 1, N(t) = 2} as {m <s<m+n2<t<m+n2+73}. You 
can compute the probability of the latter, since 71, n2, n3 are exponentially distributed 
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6.2.3 Poisson Process Starts from Scratch at Time t 


Imagine that you are to take part in an experiment to count the emissions of 
a radioactive particle. Unfortunately, in the excitement of proving the lack of 
memory property you forget about the engagement and arrive late to find that 
the experiment has already been running for time t and you have missed the 
first N (t) emissions. Determined to make the best of it, you start counting right 
away, so at time t+ you will have registered N (t+ s) — N (t) emissions. It will 
now be necessary to discuss N(¢t + s) — N(t) instead of N(s) in your report. 

What are the properties of N(t+s)—N(t)? Perhaps you can guess something 
from the physical picture? After all, a sample of radioactive material will keep 
emitting particles no matter whether anyone cares to count them or not. So 
the moment when someone starts counting does not seem important. You can 
expect N(t + s) — A(t) to behave in a similar way as N(s). And because 
radioactive emissions have no memory of the past, N(t + s) — N(t) should be 
independent of N (t). 

To study this conjecture recall the construction of a Poisson process N (t) 
based on a sequence of independent random variables 7,,72,... , all having the 
same exponential distribution. We shall try to represent N(t +s) — N(t) ina 
similar way. 

Let us put 


ni = EN(t)+1 — t, Nn =NN(t)tn, 1 = 2,3,..., 
see Figure 6.3. These are the times between the jumps of N(t+s)—N(t). Then 
we define 


En = mt +m, 
max{n: & < s}. 


h 
va) 
Dae 
I! 


I 


ew.+1 on.42 ÉN,+3 





Figure 6.3. The random variables né, n$, ná, .. 
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Exercise 6.8 


Show that 
N*(s) = N(t+ 8) — N(t). 


Hint First show that £f = Enct)in — t 


If we can show that n£, ný, ... are independent random variables having the 
same exponential distribution as 7,,72,... , it will mean that N(t +s) — N(t) 
is a Poisson process with the same probability distribution as N(s). Moreover, 
if the nt turn out to be independent of N (t), it will imply that N(t+s) — N(t) 
is also independent of N (t). 

Before setting about this task beware of one common mistake. It is some- 
times claimed that the times nj, né, n4, ... between the jumps of N(t+s)—N(t) 
are equal to £n41 — t.n+42,n43,-.. for some n. Hence nÉ, né, né,... are inde- 
pendent because the random variables En+1, Nn+2; Nn+3,... are. The flaw in this 
is that, in fact, nÉ, n$, n$, ... are equal to €,41 — t, Nn42,7n+3,--. only on the set 
{N (t) = n}. However, the argument can be saved by conditioning with respect 
to N(t). Our task becomes an exercise in computing conditional probabilities. 


Exercise 6.9 
Show that 
P{nf > s|N(t)} = P{m > s}. 
Hint It suffices (why?) to verify that 
P{m > s, N(t) =n} = P{m > s} P{N(t) = n} 
for any n. To this end, write the sets {N (t) = n} and {nj > s, N(t) = n} in terms of 


En and 7n+1, which are independent random variables, and use the lack of memory 
property for Nn+1- 


Exercise 6.10 
Show that 

P{n; > SiN > skl N (t)} = P{m > sı} ns -P{nk > Sk}. 
Hint Verify that 


P{m > 81,92 > 82,- 3k > Sk, N(t) = n} = P{m > si} ---P{me > sk}P{N(t) = n} 


for any n. This is done in Exercise 6.9 for k = 1. 
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Exercise 6.11 


From the formula in Exercise 6.10 deduce that the random variables 7}, and 
N (t) are independent, and the nf have the same probability distribution as nn, 
i.e. exponential with parameter À. 


Hint What do you get if you take the expectation on both sides of the equality in 
Exercise 6.10? Can you deduce the probability distribution of ņ,? Can you see that 
the ný are independent? 

To prove that the 7, are independent of N(t) you need to be a little more careful 
and integrate over {N(t) = n} instead of taking the expectation. 

Because N(t + s) — N(t) can be defined in terms of nj,75,... in the same 
way the original Poisson process N(t) is defined in terms of 7,72,... , the 
result in Exercise 6.11 proves the theorem below. 


Theorem 6.1 
For any fixed t > 0 
N'(s)=N(t+s)—N(t), s2>0 
is a Poisson process independent of N (t) with the same probability law as N(s). 
That is to say, for any s,t > 0 the increment N(t+s)— N(t) is independent 


of N(t) and has the same probability distribution as N(s). The assertion can be 
generalized to several increments, resulting in the following important theorem. 









Theorem 6.2 
For any 0 < tı < tg <--- < tn the increments 
N(t1), N(tz) — N(ti), N(ts) — N(t2),.-.,N (tn) — N(tn-1) 


are independent and have the same probability distribution as 


(t1). N(ta — t1), N(t3 SP pas N(tn te). 


Proof 
From Theorem 6.1 it follows immediately that each increment N (ti) — N (ti—1) 
has the same distribution as N (t; — ti—1) fori=1,...,n. 


It remains to prove independence. This can be done by induction. The case 
when n = 2 is covered by Theorem 6.1. Now suppose that independence holds 
for n increments of a Poisson process for some n > 2. Take any sequence 


NL Pa OSS Se. E E oaa 


148 Basic Stochastic Processes 


By the induction hypothesis 
N(tn41) — N(tn),...,N(t2) — N(h) 
are independent, since they can be regarded as increments of 
N"(s) = N(t, + s) — N(t1), 


which is a Poisson process by Theorem 6.1. By the same theorem these incre- 
ments are independent of N(t,). It follows that the n + 1 random variables 


N(tn41) i N (tn), nes , N (t2) a N (ti), N (tı) 


are independent, completing the proof. O 


Definition 6.7 


We say that a stochastic process £(t), where t € T, has independent increments 
if 

E(t1) =< E(to), ae » E(tn) = E(tn-1) 
are independent for any to < tı <--: < tn such that to, ti,... tn ET. 


Definition 6.8 


A stochastic process €(t), where t € T, is said to have stationary increments if 
for any s,t € T the probability distribution of ¿(t + h) — €(s + h) is the same 
for each h such that s+h,t+he T. 


Theorem 6.2 implies that the Poisson process has stationary independent 
increments. ‘The result in the next exercise is also a consequence of Theorem 6.2. 


Exercise 6.12 


Show that N(t) — At is a martingale with respect to the filtration F; generated 
by the family of random variables {N (s) : s € [0, t]}. 


Hint Observe that N(t) — N(s) is independent of F, by Theorem 6.2. 


6.2.4 Various Exercises on the Poisson Process 


Exercise 6.13 


Show that éo < 1 < 2 <::: as. 
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Hint What is the probability of the event {En-1 < En} = {nn > 0}? What is the 
probability of the intersection of all such events? 


Exercise 6.14 


Show that limy_.o En = œ as. 


Hint If limn+0€n < oo, then the sequence 7, 72,... of independent random vari- 
ables, all having the same exponential distribution, must be bounded. What is the 
probability that such a sequence is bounded? Begin with computing the probability 
P{m < m,..., n < m} for any fixed m > 0. 

Although it is instructive to estimate P {limn+oo fn < oo} in this way, there is a 


more elegant argument based on the strong law of large numbers. What does the law 
of large numbers tell us about the limit of fn as n — oœ? 


Exercise 6.15 


Show that €, has absolutely continuous distribution with density 


-ae Ab 


fn(t) = re (n = 1)! 


with parameters n and A. The density f,(t) of the gamma distribution is shown 
in Figure 6.4 for n = 2,4 and A= 1. 


igure 6.4. Density f,(t) of the 
mma distribution with parame- 
tersn=2,X\=landn=4,A\=1 





Hint Use the formula for P {En > t} in the proof of Proposition 6.1 to find the dis- 
tribution function of n. Is this function differentiable? What is the derivative? 


Exercise 6.16 


Prove that 
jim N(t)=0o as. 
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Hint What is the limit of P{N(k) > n} as k => oo? Can you express {limt+oo N (t) 
= oo} in terms of the sets {N(k) > n}? 
Exercise 6.17 
Verify that 
P{N(t) is odd} = e~% sinh(At), 
P{N(t) is even} = e~*' cosh(At). 


Hint What is the probability that N(t) = 2n +1? Compare this to the n-th term of 
the Taylor expansion of sinh At. 


Exercise 6.18 


Show that 
N(t) 


lim =) 
t— co 


if N(t) is a Poisson process with parameter A. 


Hint N(n) is the sum of independent identically distributed random variables N(1), 
N(2)—N(1),..., N(n)— N(n — 1), so the strong law of large numbers can be applied 
to obtain the limit of N(n)/n as n —> oo. Because N(t) is non-decreasing, the limit 
will not be affected if n is replaced by a continuous parameter t > 0. 


6.3 Brownian Motion 


Imagine a cloud of smoke in completely still air. In time, the cloud will spread 
over a large volume, the concentration of smoke varying in a smooth manner. 
However, if a single smoke particle is observed, its path turns out to be ex- 
tremely rough due to frequent collisions with other particles. This exemplifies 
two aspects of the same phenomenon called diffusion: erratic particle trajec- 
tories at the microscopic level, giving rise to a very smooth behaviour of the 
density of the whole ensemble of particles. The Wiener process W(t) defined 
below is a mathematical device designed as a model of the motion of individual 
diffusing particles. In particular, its paths exhibit similar erratic behaviour to 
the trajectories of real smoke particles. Meanwhile, the density fwg of the 
random variable W(t) is very smooth, given by the exponential function 
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€ 
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which is a solution of the diffusion equation 


Of 1 0? O* f 
Ot 2 dx? 
and can be interpreted as the density at time t of a cloud of smoke issuing form 
single point source at time 0. The Wiener process W(t) is also associated with 
the name of the British botanist Robert Brown, who around 1827 observed 
the random movement of pollen particles in water. We shall study mainly the 
one-dimensional Wiener process, which can be thought of as the projection of 
the position of a smoke particle onto one of the axes of a coordinate system. 
Apart from describing the motion of diffusing particles, the Wiener process 
is widely applied in mathematical models involving various noisy systems, for 
example, the behaviour of asset prices at the stock exchange. If the noise in the 
system is due to a multitude of independent random changes, then the Central 
Limit Theorem predicts that the net result will have the normal distribution, a 
property shared by the increments W(t) — W (s) of the Wiener process. This is 
one of the main reasons of the widespread use of W (t) in mathematical models. 


6.3.1 Definition and Basic Properties 


Definition 6.9 


The Wiener process (or Brownian motion) is a stochastic process W(t) with 
values in R defined for t € [0, 00) such that 


1) W(0) = 0 as.; 

2) the sample paths t+ W(t) are a.s. continuous; 

3) for any finite sequence of times 0 < tı < -::: < tn and Borel sets 
Aj,...,An CR 


P{W (t1) ) € Aj,. i ., W (tn) € An} 


= J te? (t1, 0, DY ) p (t2 — ti, 21, T2) +: 
Ai 


P (tn — tn—1,2n- 1; In) dx, dEn, 


where 1 5 
t,x, y) = =e it (6.9) 


defined for any z,y € R and t > 0 is called the transition density. 





A tyniral camnla nath af the Wiener nrocess is shown in Figure 6.5. 
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Figure 6.5. A typical path of 
W (t) 





Exercise 6.19 
Show that 


l _ 
LSS SE 


2t 
is the probability density of W (t) and find the expectation and variance of W (t). 


Hint The density of W (t) can be obtained from condition 3) of Definition 6.9 written 
for a single time t and a single Borel set. You will need the formula 


+00 a 
J e 2 dr =vVÊ2r 


OO 


to compute the integrals in the expressions for the expectation and variance. 


Remark 6.1 
The results of Exercise 6.19 mean that W(t) has the normal distribution with 


mean 0 and variance t. 


Exercise 6.20 


Show that 
E (W(s)W(t)) = min {s,¢}. 


Hint The joint density of W(s) and W(t) will be needed. It can be found from 
condition 3) of Definition 6.9 written for two times s and t and two Borel sets. 


Exercise 6.21 


Show that 
E (I(t) > W (s)|?) = jt- sl]. 
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Hint Expand the square and use the formula in Exercise 6.20. 


Exercise 6.22 
Compute the characteristic function E (exp («AW (t))) for any A € R. 


Hint Use the density of W(t) found in Exercise 6.19. 


Exercise 6.23 
Find E (W(t)4). 


Hint This can be done, for example, by expressing the expectation in terms of the 
density of W(t) and computing the resulting integral, or by computing the fourth 
derivative of the characteristic function of W(t) at 0. The second method is more 
efficient. 


Definition 6.10 


We call W(t) = (W'(t),...,.W"(t)) an n-dimensional Wiener process if 
Wi (t),...,W”(t) are independent R-valued Wiener processes. 


Exercise 6.24 


For a two-dimensional Wiener process W(t) = (W1(t),W7?(t)) find the prob- 
ability that |W(t)| < R, where R > O and |z| is the Euclidean norm of 
a = (z!, x?) in R, ie. |z| = (x) ° + (2)*. 


Hint Express the probability in terms of the joint density of W'(t) and W(t). In- 
dependence means that the joint density of W+ (t) and W(t) is the product of their 
respective densities, which are known from Exercise 6.19. It is convenient to use polar 
coordinates to compute the resulting integral over a disc. 


6.3.2 Increments of Brownian Motion 


Proposition 6.2 


For any 0 < s < t the increment W(t) — W (s) has the normal distribution with 
mean 0 and variance t — s. 
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Proof 
By condition 3) of Definition 6.9 the joint density of W (s), W(t) is 


Fw(s),w(t) (x,y) = p(s,0,zr) p(t —s,2z,y). 


Hence, for any Borel set A 


P{W) Wis) eA} = | p(s,0,2) p(t — s,2,y) dz dy 
{(z,y):y-zEA} 


+00 
= J p(s,0,z) (/ p(t- s,z,y) a) dr 
—0o {y:y—-rEA} 
+00 
= J p (s,0, x) (f pt-szz+u du ) dz 
— o0 A 


+00 
= f p(s,0, x) (f pt- 5.0.0) iu) dx 
— Oo A 


+00 
= | ptt-s,0,u) du | p(s,0, x) dz 
A 


= J pë-s,0,u) du. 
A 


But f (u) = p(t — s,0,u) is the density of the normal distribution with mean 
0 and variance t — s, which proves the claim. O 


Corollary 6.1 


Proposition 6.2 implies that W (t) has stationary increments. 


Proposition 6.3 
For any 0 = tọ < tı < --- < tn the increments 
W (tı) gi W (to), aera W (tn) = W (tn—1) 


are independent. 


Proof 


From Proposition 6.2 we know that the increments of W(t) have the normal 
distribution. Because normally distributed random variables are independent 
if and only if they are uncorrelated, it suffices to verify that 


E [(W (u) — W(t) (W(s) - W(r))] = 0 
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for any 0< r < s< t< u. But this follows immediately from Exercise 6.20: 


E [(W (u) — W (t)) (W (s) — W (r))] = E (W (u)W(s)) - E (W (u)W (r)) 
— E(W(t)W (s)) + E (W (t)W (r)) 


SS aT 
0, 


as required. O 


Corollary 6.2 


For any 0 < s < t the increment W(t) — W (s) is independent of the o-field 


F, =0{W(r):0<r< s}. 


Proof 


By Proposition 6.3 the random variables W (t)—W (s) and W (r)-W (0) = W(r) 
are independent if 0 < r < s < t. Because the o-field F, is generated by such 
W (r), it follows that W(t) — W (s) is independent of F.. 0 


Exercise 6.25 


Show that W(t) is a martingale with respect to the filtration F+. 


Hint Take advantage of the fact that W(t) — W (s) is independent of Fs if s < t. 


Exercise 6.26 
Show that W(t)? — ¢ is a martingale with respect to the filtration F;. 
Hint Once again, use the fact that W(t) — W (s) is independent of F, if s < t. 


Let us state without proof the following useful characterization of the 
Wiener process in terms of its increments. 


Theorem 6.3 


A stochastic process W (t), t > 0, is a Wiener process if and only if the following 
conditions hold: 


1) W(0) = 0 a.s.; 


2) the sample paths t 4 W (t) are continuous a.s.; 
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3) W(t) has stationary independent increments; 


4) the increment W(t) — W (s) has the normal distribution with mean 0 and 
variance t — s for anyO0<s<t. 


Exercise 6.27 
Show that for any T > 0 

V(t) =W(t+T)-W(T) 
is a Wiener process if W(t) is. 


Hint Are the increments of V(t) independent? What is their distribution? Does V(t) 
have continuous paths? Is it true that V(0) = 0? 


The Wiener process can also be characterized by its martingale properties. 
The following theorem is also given without proof. 


Theorem 6.4 (Lévy's martingale characterization) 


Let W(t),t > 0, be a stochastic process and let Fy = o(W,,s < t) be the 
filtration generated by it. Then W(t) is a Wiener process if and only if the 
following conditions hold: 


1) W(0) = 0 a.s.; 
2) the sample paths t => W (t) are continuous a.s.; 
3) W(t) is a martingale with respect to the filtration F;; 


4) WA — t is a martingale with respect to F;. 


Exercise 6.28 
Let c > 0. Show that V(t) = +W (c°t) is a Wiener process if W (t) is. 


Hint Is V(t) a martingale? With respect to which filtration? Is |V (t)|? — t a martin- 
gale? Are the paths of V(t) continuous? Is it true that V(0) = 0? 


6.3.3 Sample Paths 


Let 
Gat <4, <2 <7, HT, 
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- where 

iT 

n } 

be a partition of the interval [0,7] into n equal parts. We denote by 


N oo 
t? = 


AW = W (tip) — Wty) 


the corresponding increments of the Wiener process W(t). 


Exercise 6.29 
Show that 


n—l 


: n 2. : 2 
SP al =T in L*. 


Hint You need to show that 
n—-1l 2 
; n 2 PNS 
lim E (S (ALW) r ) = 0. 


Use the independence of increments to simplify the expectation. What are the expec- 
tations of ATW, (ALW)? and (A? W)*? 


The next theorem on the variation of the paths of W(t) is a consequence of 
the result in Exercise 6.29. First, let us recall that the variation of a function 
is defined as follows. 


Definition 6.11 


The variation of a function f : [0,7] —> R is defined to be 


n—1 


eae 2 fti) — fti), 


where t = (to, t1, ..., tn) is a partition of [0, T], i.e. 0 = to < ti <- <tn =T, 
and where 


At= max [tsa = t;| : 
7=-0,...,.n—1 


Theorem 6.5 


The variation of the paths of W(t) is infinite a.s. 
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Proof 
Consider the sequence of partitions t” = (tj,t?,...,t") of [0,7] into n equal 
parts. Then 
n—1l n—l 
nggi? n n 
D larw < (__max lari) Yo iarw 
i= = 


Since the paths of W(t) are a.s. continuous on [0, T], 


lim (mex lawi) =0 a.s. 


n— Oo =0 502,71 
By Exercise 6.29 there is a subsequence t”! = (to*,t)*,...,t7*) of partitions 
such that 
Nnk—i 
lim AW]? =T as. 
ža $ arw 


This is because every sequence of random variables convergent in L? has a 
subsequence convergent a.s. It follows that 


nkg—l 
lim ) |A; W| = a.s., 
k= <4 0 

= 


while 


T 
lim At”: = lim — = 0, 
k= o0 k= NE 


which proves the theorem. O 


Theorem 6.5 has important consequences for the theory of stochastic inte- 
grals presented in the next chapter. This is because an integral of the form 


T 
f f(t) aw (t) 


MS 


cannot be defined pathwise (that is, separately for each w € N) as the Riemann- 
Stieltjes integral if the paths have infinite variation. It turns out that an intrinsi- 
cally stochastic approach will be needed to tackle such integrals, see Chapter 7. 


Exercise 6.30 
Show that W (t) is a.s. non-differentiable at t = 0. 
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Hint By Exercise 6.28 V.(t) = +W (c't) is a Wiener process for any c > 0. Deduce 
that the probability 


P { l > cM for some t € (0, 4]} 
is the same for each c > 0. What is the probability that the limit of wet exists as 
t N 0, then? 
Exercise 6.31 


Show that for any t > 0 the Wiener process W(t) is a.s. non-differentiable at t. 


Hint Vi(s) = W(s +t) — W(t) is a Wiener process for any t > 0. 


A weak point in the assertion in Exercise 6.31 is that for each t the event of 
measure 1 in which W (t) is non-differentiable at t may turn out to be different 
for each t > 0. The theorem below, which is presented without proof, shows 
that in fact the same event of measure 1 can be chosen for each t > 0. This is 
not a trivial conclusion because the set of t > 0 is uncountable. 


Theorem 6.6 
With probability 1 the Wiener process W(t) is non-differentiable at any t > 0. 


6.3.4 Doob’s Maximal L* Inequality for Brownian Motion 


The inequality proved in this section is necessary to study the properties of 
stochastic integrals in the next chapter. It can be viewed as an extension of 
Doob’s maximal L? inequality in Theorem 4.1 to the case of continuous time. 
In fact, in the result below the Wiener process can be replaced by any square 
integrable martingale €(t), t > 0 with a.s. continuous paths. 


Theorem 6.7 (Doob’s maximal L? inequality) 


For any t > 0 
E (maxw) < 4E|W(t)|’. (6.6) 
gS 


Proof 
For t > 0 and n € N we define 


up =|w(#)|, osre 07 
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Then, by Jensen’s inequality, My;,k = 0,---,2”, is a non-negative square in- 
tegrable submartingale with respect to the filtration Ff = F kt, SO by Theo- 


rem 4.1 
E ( max IMPI?) < 4EM = 42) (0)? 


Since W (t) has a.s. continuous paths, 


lim max |M?|? = max|W (s)|? a.s. 
n= k<2" s<t 


Moreover, since M? = Mee the sequence sup,.<on |M p| n € N, is increasing. 
Hence by the Lebesgue monotone convergence theorem maxs<t |W (s)|? is an 
integrable function and 


7 2 Pp a ni2 2 
B (max|W(s))?) = lim E (max imei?) < 4B? 


completing the proof. O 


6.3.5 Various Exercises on Brownian Motion 


Exercise 6.32 
Verify that the transition density p(t, x,y) satisfies the diffusion equation 


Op 10°p 


ðt - 2 By?’ 
Hint Simply differentiate the expression (6.5) for the transition density. 


Exercise 6.33 
Show that Z(t) = —W (t) is a Wiener process if W(t) is. 


Hint Are the increments of Z(t) independent? How are they distributed? Are the 
paths of Z(t) continuous? Is it true that Z(0) = 0? 


Exercise 6.34 
Show that for any 0<s<t 


P{W(t) € AlW(s)} = f. p (t — s, W (8), y) dy. 
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‘Hint Write the conditional probability as the conditional expectation of 14(W(t)) 
given W(s). Compute the conditional expectation by transforming the integral of 
14(W(t)) over any event in the o-field generated by W (s). This can be done using 
the joint density of W(s) and W(t). Refer to the chapter on conditional expectation 
if necessary. 


Exercise 6.35 


Show that eW(e-2 is a martingale. (It is called the exponential martingale.) 


Hint What is the expectation of e”“)~”*) for s < t? By independence it is equal 
to the conditional expectation of ev = Wt) given Fs. This will give the martingale 
condition. 


Exercise 6.36 
Compute E (W(s)|W(t)) forO<s<t. 


Hint You want to find a Borel function F such that E (W(s)|W(t)) = F (W(t)), i.e. 


J W(s)dP = / F (W (t)) dP. 
{Ww (tye A} | {Ww (t)eA} 


Either side of this equality can be transformed using the joint density of W(t) and 
W (s). 


6.4 Solutions 


Solution 6.1 


Suppose that 7 is a random variable with exponential distribution of rate A. 
The distribution function of 77 is 


0 if t < 0, 


PO =Pinst}=1-P9>8={ 9 e So 


Therefore 7 has density 


d 0 if t < 0, 
MO) = GPO =4 Net if t > 0. 


The distribution function F(t) and density f(t) are shown in Figure 6.6. 


Solution 6.2 


Tloine the doncitw f(t) = \e—A* fonnd in Exercise 6.1 and inteeratine bv parts. 
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Figure 6.6. The distribution function F(t) and density f(t) of a random variable 
with exponential distribution of rate À = 2 


we obtain 


OO OO OO d 
E(n) = J tf(t)dt = De dt = -| tres dt 
0 


In a similar way we compute 


E(n’) oF t? f(t) d t= fo t? re ** dt = -f poe dt 


ü 2 
2 —Àt ert = 
= —t“e N +2 te dt = 0+ f as f(t) dt = 55. 


It follows that the variance is equal to 
var (n) = E (n?) - (E M = 53 - 3 = 


Solution 6.3 


By the definition of a random variable with exponential distribution 


P {q > s +t} =e At) = e-Me-*M = P {n > 8} P{n> t}. 


Solution 6.4 
By the definition of conditional probability 
P{n>t+s,n> s8} 
P {1 > 8} 
_ P{n>t+s} 
= P{n>s} ’ 
since e >t+s,n>s}={n> t + s} (because ņ > s +t implies that 7 > s). 


RYsct. 82 2 abe hee PP PN te Ate lant ta (4A OY 


' 
J 


P {n >t+ sin >s} 
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- Solution 6.5 


The exponential distribution is the only probability distribution satisfying the 
lack of memory property because the only non-negative non-increasing solutions 
of the functional equation 


g(t + s) = g(t)g(s) 
are of the form g(t) = a’ for some 0 <a < 1. 
To verify this observe that [g(m/n)]” = g(m) = [g(1)]™ for any integers m 
and n Æ 0. Let a := g(1). It follows that 
g(q)=a!’ foranygeEeQ 


Since g is non-increasing, 0 <a < 1 and 
t . t 
a= inf > g(t) > su =i sg 
PL set ie Sup 9) 
so indeed 
g(t)=a’ foranyteR 
As a result, P{n > t} = a‘ for some 0 < a < 1. But the distribution function 
of a random variable cannot be constant, so 0 Æ a Æ 1. Hence a = e~®* for some 
A > 0, completing the argument. 
Solution 6.6 


Since N(¢) has the Poisson distribution with parameter At we have E (N(t)) = 
At. Indeed 


Yo nP iN = =n; = vs a 


n=0 


E (N(t)) 


_ -At (At)” ENA 
= Ate Daly e TS AL 





Solution 6.7 


Using the fact that 71,72,... are independent and exponentially distributed, 
we obtain 


P{N(s) =1,N(t) = 2} 


P{&i <s <é <t< &} 
P{m <s<mt+m<t<m+72+73} 


S 
f P{s<utm <t<u+m +n }àe™™du 
0 


| 


s t—u 
[ ( | P{t<u+v+n} red ) Ne >" du 
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S t—u 
/ (J Mw Ae dy) he *“ du 
0 s—u 


A eT% s (t — 8). 


Solution 6.8 


Since 
En = Mi tta 

oe EN(t)+1 aes NN(t)+2 + °°" + TN(t)+n 

= En(t)in — t, 
it follows that 
max{n: éé < s} 
max{n : Evin St +9} 
max{n:&, <t+s}-— N(t) 
N(t+s) — N(t). 


N*(s) 


II 


Solution 6.9 
It is easily verified that 


{N(t)=n} = {n41 >t—- En, t > En} 
{ni >s,N(t) =n} a {n41 > St- En t> En} 


Since n, nn+1 are independent and nn+1ı satisfies the lack of memory property 
from Exercise 6.3, 


P {n+ >st+t—&,t>&} 


t 
J P {n41 > s +t -— u} Pe, (du) 


P {ni >s,N(t) =n} 


P {141 > 9f P {m+ > t — u} Pe, (du) 
P {n41 > s} P {nn+1 zis Ent 2 En} 

P {m41 > s} P{N(t) =n} 

P{m > s}P{N(t) =n}. 


The last equality holds because nn41; has the same distribution as 7,. Now 
divide both sides by P{N (t) = n} to get 


P{ni > s|N(t) =n} = P{m > $} 
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for any n = 0,1,2,.... As a result, 
P{n, > s|N(t)} = P{m > s} 


because N (t) is a discrete random variable with values 0,1,2,.... 


Solution 6.10 
As in Solution 6.9, 
{N(t) =n} = {m+ >t —-&n,t > én}, 
{ni > sı, N(t) =n} = {n41 > 81 +t- ént > ên} 
and, more generally, 
{ni > $1,12 > $2,---,N > Sk, N(t) =n} 
= {M41 > 81 +t — En, t > En} N {M42 > S2} NNA {nk > Sk} 


Since En, Nn+1,---;Nn+k are independent and Mn+2,...,m+k have the same 
distribution as n2,...,7,, using Exercise 6.9 we find that 


P{ni > 81,75 > $2,---. > sk, N(t) =n} 

= P{tn+i > $1 +t - En, En <t}P{rnte > s2} P{ūm+k > sk} 
P{nj > sı, N(t) = n}P{n2 > 82}---P{e > sx} 

P{n\ > si|N(t) = n}P{n2 > s2}: P{nk > sp} P{N(t) =n} 
P{m > 81}P{n2 > s2}---P{ > sp }P{N(t) =n}. 

As in Solution 6.9, this implies the desired equality. 


Solution 6.11 


Take the expectation on both sides of the equality in Exercise 6.10 to find that 
© P{yi > $1,---,% > sk} = P{m > s1}: P{Nk > Sk}. 
If all the numbers sn except perhaps one are zero, it follows that 
P{n}, > sn} = Pin > on}, n= 1,...,k, 


so the random variables n have the same distribution as nn. Inserting this 
back into the first equality, we obtain 


P{n > 81,..-,m > sk} = P{ny > si} ---P{ni > se}, 


so the nf are independent. 
To prove that the n‘, are independent of N(t) integrate the formula in 
Exercise 6.10 over {N(t) = n} and multiply by P{N (t) = n} to get 


P{ni > 81,..-,% > Sk, N(t) =n} = P{m > s1} P{nk > sp }P{N(t) =n}. 
But P{nt > sn} = P{mn > sn}, hence N(t) and the nt are independent. 


I 


| 
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Solution 6.12 


We need to verify conditions 1), 2), 3) of Definition 6.3. Clearly, N(t) — At is 
F,-measurable. By Exercise 6.6 


E (|N(t)|) = E (N(t)) = At < œ, 


which means that N (t) is integrable, and so is N(t) — At. 
Theorem 6.2 implies that N (t)—N (s) is independent of F, for any 0 < s < t, 
SO 


E(N(t) — N(s)| Fs) = E(N(t) — N(s)) = E(N(t)) — E(N(s)) = At — As. 
It follows that 
E(N(t) — At|F,) = E(N(s) — As|Fs) = N(s) — As, 
completing the proof. 


Solution 6.13 
Since Nn = En — €n—1 and P{n, > 0} = e® = 1, 


P{&o < a <e< I= P(Aim>0] =]: 


n=l 
Here we have used the property that if P(A,) = 1 for all n = 1,2,... , then 
P Ay) = 1, 
Solution 6.14 
Since 


OO 
aim, & = Dts 
n=l 
it follows that 


{ lim én < 00} C {,72,... is a bounded sequence} 
n— CO 
CO OO 
= J () {n < M}. 
ma=aln=i 


Let us compute the probability of this event. Because An {h <m}, N= 
1,2,... is a contracting sequence of events, 


ore N 
p(Ñ) = j P (Nim <m) 


n=1 n=1 
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N 
jim, II P{nn < m} 


: _ Àm N 
lim (1 — e^") 


0. 


| 


It follows that 


P ( lim €n < 0) < msm) 


A 
Y 
ET 


Ca 
8 


— 

z 
il 
— 


{tn < mi) 


lA 
Ms: 

ty 
ae 
__)8 


3 
UJ 
3 
UL 


lI 
© 


completing the proof. 
While it is instructive to work through the above estimates, there exists a 
much more elegant argument. By the strong law of large numbers 


lim Èn =~ as 
noo n 
Here x is the expectation of each of the independent identically distributed 


random variables n, (see Exercise 6.2). It follows that 


him. :€,:== 00. 2:5. 
N—> OO 


as required. 


Solution 6.15 


In the proof of Proposition 6.1 it was shown that 


P {6n > t} =e ag — 


for t > 0, see (6.4). Therefore the ern esti | 


oc k 
Falt) = P{& St} = 1- P En > t) = e~ yy A 


k=n 


of En is differentiable, the density fn of En being 
falt) = = Fn(t) 


— —\ent eg cod 4 \enm™ 3 a 


—At ai" 
(n — 1)! 





= Àe 
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for t > 0, and clearly f,(t) = 0 for t < 0. 
Solution 6.16 


Because N(t) has non-decreasing trajectories 


f lim N(t) = o0} = N JANG) > n}. 
n=1k=1 


Also, {N(k) > n}, k = 1,2,... is an expanding sequence of events and 


P{N(k) > n} = e> 3 old 


n—-l i 
— po S e i as k —> oo. 


i=0 
[t follows that = 
P { Uo > n)} =1, 
k=1 
30 a 
P f lim N(t) = œ} -Pif L_) {N(k) > n)} = 
n=l k=1 
Solution 6.17 
Since 
et — et ond enti 
inh T Se ee pens eee 
en 2 2 (2n + 1)!’ 
eT tet SY r?’ 
cosh(z) = 5 = > ni 
ve have 


P{N(t) is odd} = 3 P{N(t) = 2n +1} 


n=0 


m 5 oot (At)?rtt 
med (2n + 1)! 


= e *sinh(At), 


poaae 


P{N(t) is even} = 5 P{N(t) = 2n} 


n=0 


n (2n)! 
e~t cosh(At). 


m 3 oot (At)?” 
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Solution 6.18 
We can write 
N(n) = N(1) + (N(2) - N(1)) + + (N(n) - N(n - 1)), 
where N(1), N(2) — N(1), N(3) — N(2),... is a sequence of independent iden- 
tically distributed random variables with expectation 
E(N(1)) = E (N(2) — N(1)) = E (N(3) — N(2)) =: =A. 
By the strong law of large numbers 


lim INAR) 


n— Oo n 
Now, ifn < t< n+ 1, then N(n) < N(t) < N(n +1) and 
N(n) | N(t) < Nint)) 
—. 


SEN ANA 
n+1 -~ n 


=X as. (6.8) 





By (6.8) both sides tend to À as n —> oo, implying that 


lim A 


too t 


Solution 6.19 
Condition 3) of Definition 6.9 implies that 
fwe) (x) = p(t, 0, x) 


is the density of W(t). Therefore, integrating by parts, we can compute the 
expectation 


+00 
E (W(t) f xp (t,0, x) dz 





1 ToS 2? 
= —— re © dr 
V 2rt J- 
t ii E 
= —-—— —e 2 dr 
V2rt Jo dT 
t 2 | +00 
= — e 2t = 
V 2t —0o 
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nd variance 








+00 
E((w(t))’) = J r?’ p(t,0, £) dz 
1 TOR 22 
= = | re % dx 
V “Tl J-œ 
t + 00 d -£ 4 
= — -e —_— 2t 
t „2jto t T zd 
= — — ~ 2t + e 2t dx 
t ey ee 
= 0+ — e 2? du=t 
WS 21 J =e 


Ve have used the substitution u = Ti and the formula stated in the hint. 


olution 6.20 


uppose that s < t. Condition 3) of Definition 6.9 implies that the joint density 
f W(s) and W(t) is 


fw(s),w(t) (x,y) ag p(s,0,z)p{t = S, £, Y) : 


follows that 


+00 +00 
E(W(s)W(t)) = f [ ry p(s,0,z) p(t—s,z,y) dz dy 


+00 +00 
=| xp(s,0,2) ( f yp(t~s,2,y) dy dx 


+00 
= J z’p(s,0, x£) dr = s. 


— 00O 


his is because by the results of Exercise 6.19 


‘ioe +00 
yp(t—s,z,y) dy =| (z+u)p(t—s,x,x +u) du 
—00 — 00 
+00 | 
z. (x +u) p(t—s,0,u) du 
Ta +00 
= | p(t — s,0,u) au | up(t—s,0,u)} du 
= 7 PSs 
id 


+00 
f r’p(s,0, x) dz = 8. 
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-It follows that for arbitrary s,t > 0 
E (W(s)W(t)) = min {s, t}. 
Solution 6.21 


Suppose that s < t. Then by Exercise 6.20 


E (Iw) = W(s)|) = E(W(t)?) —2E(W(s)W(t)) + E (W(s)°) 
= t-—2s+s=t-s. 


In general, for arbitrary s,t > 0 
E (IW (e) = W(s)|?) =|t—s|. 


Solution 6.22 
Using the density fw t) (x) = p (t,0, x) of W(t), we compute 








+00 
E (exp ((AW (t))) = J e7n(t,0,2) dz 
— OO 
ee, ees 
= eTe 2 dx 
V ant I. 
T eT [~ e” sh dx 
JV 27t — 00 
= e 


Solution 6.23 


Using the formula for the characteristic function of W (t) found in Exercise 6.22, 
we compute 

dé 
d4 
aa 
dàt lA=0 


= 3t?. 


E (W()!) = 





E (exp (iAW (t))) 
0 


A= 


2 
eat 


Solution 6.24 


Since W1(t), W?(t) are independent, their joint density is the product of the 
densities of W!(t) and W*(t). Therefore 


Pwo <R} = f 


p (t,0, x) p(t,0,y) dz dy 
{lzi<R} 
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1 z?4y? 
= — e ~ 3t dr dy 


ant J{jz|<R} 


2n 
= -5 
a Lf J re dip dr 
=: ee o O-& 
[4 ae dr 


2 
=l-e ®. 


'e have used the polar coordinates R, y to compute the integral. 


olution 6.25 
wanyO<s<t 
E(W(t)|Fs) = E (W(t) —-W(s)|Fs) + E (W (s)| Fs) 
E (W(t) — W(s)) + W(s) 
= W(s), 
nce W (t)—W (s) is independent of F, by Corollary 6.2, W (s) is #,-measurable 
id E(W(t)) = E(W(s)) = 


olution 6.26 
wanyO<s<t 
E(W(t)|F,) = E (IW) = W(s)[? (Fe) + E (2W(t)W(s)|Fe) 
-E (W(s)?|F.) 
= E (W(t) - W (s)|?) + 2W(s)E (W(t)|F.) 


-W (s)? 
= t—s+2W(s)? — W(s)? 
= t—s+W(s)’, 


nce W(t) — W (s) is independent of F, and has the normal distribution with 
ean 0 and variance t — s, W (s) is ¥,-measurable, and W (t) is a martingale. 


follows that 
E (W(t)? -t|F,) = W(s)? — s, 


; required. 
olution 6.27 
or any 0 < tp < tı <-:: < tn the increments 
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of V(t) are independent, since the increments 
W (th +T) —-W(tn-14+T),:--,W(ti +T) —-W(to +T) 


of W(t) are independent. For any 0 < s < t the increment V(t) — V (s) has the 
normal distribution with mean zero and variance t—s, since W(t+T)—-W(s+T) 
does. Moreover, the paths try V(t) = W(t+T) — W(T) are continuous and 


V(0) = W(T) —- W(T) = 0. 
By Theorem 6.3 V(t) is a Wiener process. 


Solution 6.28 
It is clear that V(0) = +W (0) = 0 a.s. and the paths t |} V(t) = ¿W (et) are 


a.s. continuous. We shall verify that V(t) and |V (t)|? — t are martingales with 


respect to the filtration 


Ge = o{V(s):0<s<t} 

a {W(c*s):0<s<t} 
a {W(s):0<s<c*t} 
= Jop 


Indeed, if s < t, then c?s < c*t, so 


E (VDG) = B(LW(eH Fan) 
5 1E (W (c2t)|Fe2,) 
= <W(cs) =V(s) 

and 

B (VOP - 1g) = E (Z WED -tF ) 

= ZE (|W(et)|’ - tF) 
- 1 (west -es) 

IV(s)? =s, 


since W(t) and |W (t)|? — t are martingales with respect to the filtration F,. It 
follows by Levy’s martingale characterization that V(t) is a Wiener process. 
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Solution 6.29 
Since the increments A?W are independent and 


B(atw)=0, B((atw))=2, B((apwy') i 


it follows that 














n—1 ? am l 
e( P(w) -T ) e( > (am E 
-Zefa 
i > [2 (am t) - Ze (arw?) + F] 
E Oa a 
as n —> oo. 


Solution 6.30 


We claim that, with probability 1, for any positive integer n there is a t € [0, 2] 
such that [ww > n. This condition implies that W (t) is not differentiable at 
t= 0. 
Let us put 
A, = { ol > n for some t € (0, 2)} 


By Exercise 6.28 n 
_ 4 
Va(t) = SW (n't) 
is a Brownian motion for any n. Therefore 
ni 
P l wom > n} 
p{ en ) x n} 


P{|W(1)|>4}51 asn > o. 


P (An) 


IV 


Since Aj, Á2,... is a contracting sequence of events, 


P (A An) = lim P(An) =1 
n=1 


which proves the claim. 
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- Solution 6.31 


By Exercise 6.27 Vi(s) = W(s + t) — W(t) is a Wiener process for any t > 0. 
Therefore, by Exercise 6.30 Vi(s) is a.s. non-differentiable at s = 0. But this 
implies that W(t) is a.s. non-differentiable at t. 


Solution 6.32 





Differentiating 
p(t, z, y) = =e 
7 3 = e i j 
á V 2rt 
we obtain 
2 2 
y“ — 2yr + 2° -t 
gP Ty) = a5 UE), 
ð r-y 
gy 29) = ——~ ptt, z, y), 
8? y? — 2yr +r? — t 
gy7? By) = gg DE ey) 
SO 
Op  106%p 
Ot 28y?’ 
as required. 


Solution 6.33 


Clearly, Z(t) = —W(t) has a.s. continuous trajectories and Z(0) = —W(0) = 
0 as. If W(t) has stationary independent increments, then so does Z(t) = 
—W (t). Finally, 

Z(t) — Z(s) = — (W (t) — W (s)) 


has the same distribution as W (t)— W (s), i.e. normal with mean 0 and variance 
t — s. By Theorem 6.3 Z(t) is a Wiener process. 


Solution 6.34 
Let 0 < s < t. Then 


/ 1 a (W (t)) dP 
{W(s)EB} 


P{W(s) € B,W(t) € A} 


J J p(s,0,x)p(t — s, x,y) dz dy 
BJA 


[ (/ p(t — s,2,y) dy) p(s, 0,2) dx 


= [ _ ( f vte-s,W6),u ay) dP 


tl 
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or any Borel set B C R. It follows that 


P{W(t) € A|W(s)} = E (1a(W(t))|W(s)) = f p(t — s, W (8), y) dy. 


solution 6.35 


Ne shall prove that eW(e-3 isa martingale with respect to the filtration F;. 
Jlearly, it is adapted to the filtration F;, since W(t) is. Let 0 < s < t. Because 
V(t) — W (s) is independent of F, and W (s) is ¥,-measurable, 


E (eP OF.) = E (eWO-W)eW)/g, ) 
eW) E (E) 


— eW(s) E (Pom) 


[he increment W(t) — W (s) has the normal distribution with mean 0 and 
rariance t — s, so the expectation of eW()-W(s) is equal to 


+0o 
E Ca = f e* p(t — s,0, x) dz 


naa +00 
= | p(t — s,0,x — t) dx 


— OO 


t follows that 
E (c™e-4|F, ) = eWl) e$, 


t also follows that eW()e—2 is integrable. Therefore eW (© e-+ is a martingale. 


Solution 6.36 


set 0 < s < t. We are looking for a Borel function F such that E (W(s)|W(t)) = 
F (W(t)), i.e. 


J W (s) dP = j F(W(t)) dP 
{W(t)EA} {W(t)EA} 


or any Borel set A in R. The integral on the right-hand side can be written as 
J| FWO ap =f Fpl oyd 
{W{(t)E€A} A 
and the integral on the left-hand side as 


[ W(s)dP = f T aeaa de | dy 
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- using the expression for the joint density of W (s) and W(t) in Solution 6.20. 
Let us compute the inner integral: 


+ 0o OO t— 
J xp(s,0,x)p(t E S, T, Y) dx = p(t, 0, v) f a (== *y,2] dx 


—oo — 00O 


S 
= zY P(t, 0,9). 


(To see that the first equality holds, just use formula (6.5) for p(t, x, y).) There- 


fore 
S 
J W(s)aP = | zY P(t, 0,9) dy. 
{W(t)EA} A 


It follows that F(y) = fy, i.e. 


E(W(s)|W(t)) = ŻW (t). 


OEEO N EOTIERRO: Sr TE OETAN ERNETEN Y AAT REEE o EEEE ETE e e EEE EET EEEE E TEENE EET A TTA TNEI: AAEE -oiedaiennineloaioeneeieiaiadeniaa meaiaaieaiemnatenean 
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It6 Stochastic Calculus 


One of the first applications of the Wiener process was proposed by Bachelier, 
who around 1900 wrote a ground-breaking paper on the modelling of asset 
prices at the Paris Stock Exchange. Of course Bachelier could not have called 
it the Wiener process, but he used what in modern terminology amounts to 
W(t) as a description of the market fluctuations affecting the price X(t) of 
an asset. Namely, he assumed that infinitesimal price increments dX(t) are 
proportional to the increments dW (t) of the Wiener process, 


dX(t) = o dW (t), 


where ø is a positive constant. As a result, an asset with initial price X (0) = z 


would be worth 
X(t) =xzr+0W (t) 


at time t. This approach was ahead of Bachelier’s time, but it suffered from 
one serious flaw: for any t > 0 the price X(t) can be negative with non-zero 
probability. Nevertheless, for short times it works well enough, since the prob- 
ability is negligible. But as t increases, so does the probability that X(t) < 0, 
and the model departs from reality. 

To remedy the flaw it was observed that investors work in terms of their 
potential gain or loss dX (t) in proportion to the invested sum X (t). Therefore, 
it is in fact the relative price dX (t)/X (t) of an asset that reacts to the market 
fluctuations, i.e. should be proportional to dW (t), 


dX(t) = o X (t) dW (t). (7.1) 
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What is the precise mathematical meaning of this equality? Formally, it resem- 
bles a differential equation, but this immediately leads to a difficulty because 
the paths of W(t) are nowhere differentiable. A way around the obstacle was 
found by Itô in the 1940s. In his hugely successful theory of stochastic integrals 
and stochastic differential equations Itô gave a rigorous meaning to equations 
such as (7.1) by writing them as integral equations involving a new kind of 
integral. In particular, (7.1) can be written as 


X(t)=a+o0 [ X(t) dW(t), 


where the integral with respect to W(t) on the right-hand side is called the Jté 
stochastic integral and will be defined in the next section. While at first sight 
one would expect the solution to this equation to be ze), in fact it turns 


out to be 
X(t) = ceW%e-2, 


which is the exponential martingale introduced in Exercise 6.35. The intrigu- 
ing additional factor e~2 is due to the non-differentiability of the paths of the 
Wiener process. Clearly, if x > 0, then X(t) > 0 for all t > 0, as required in 
the model of asset prices. In the following sections we shall learn how to trans- 
form and compute stochastic integrals and how to solve stochastic differential 
equations. | 

Throughout this chapter W(t) will denote a Wiener process adapted to a 


filtration F; and L’ will be the space of square integrable random variables. 


7.1 Itô Stochastic Integral: Definition 


We shall follow a construction resembling that of the Riemann integral. First, 
the integral will be defined for a class of piecewise constant processes called 
random step processes. Then it will be extended to a larger class by approxi- 
mation. | 

There are, however, at least two major differences between the Riemann 
and It6 integrals. One is the type of convergence. The approximations of the 
Riemann integral converge in R, while the It6 integral will be approximated by 
sequences of random variables converging in L°. The other difference is this. 
The Riemann sums approximating the integral of a function f : [0,7] —> R are 
of the form 


>> F(ss (tsa — ty), 
j=0 
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-where 0 = to < tı < -> < tn =T and sj is an arbitrary point in [t;,t;+41] for 
each j. The value of the Riemann integral does not depend on the choice of the 
points s; € [t;,tj;41). In the stochastic case the approximating sums will have 
the form 


n—l 


>= f (83) (W (tj41) — W (t;)). 


j=0 
It turns out that the limit of such approximations does depend on the choice of 
the intermediate points s; in [t;,t;41]. In the next exercise we take f(t) = W(t) 
and consider two different choices of intermediate points. 


Exercise 7.1 


Let 0 = tọ <t? < = < ta =T, where t? = L, be a partition of the interval 
[0, T] into n equal parts. Find the following limits in L?: 


n—l1 


lim JO W(P) (W(t) — WEP) 
j=0 


and 


n—1 
lim DoW (ha) (WG) = WED). 
j=0 


Hint Apply Exercise 6.29. You will need to transform the sums to make this possible. 
The identities 


bol Re bole 


a(b—a) = (? - a?) - 5 (a0), 


b(b — a) 


(b? — a?) + 5 (a — b)? 
may be of help. 


The ambiguity resulting from different choices of the intermediate points s; 
in each subinterval [t;,tj+1] can be removed by insisting that the approxima- 
tions of the integrand should consist only of processes adapted to the underlying 
filtration F. This amounts to taking s; = t; for each j. The choice is motivated 
by the interpretation of F: the value of the approximation at t may depend 
only on what has happened up to time t, but not on any future events. 


Definition 7.1 


We shall call f(t),t > 0 a random step process if there is a finite sequence 
of numbers 0 = tọ < tı < ... < tn and square integrable random variables 
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No, 1,---;Nn—1 such that 


n—-1 


FA =o mle, tan, (7.2) 
j=0 


where 7; is F: -measurable for j = 0,1,...,n — 1. The set of random step 
processes will be denoted by MZ... 


Observe that the assumption that the n; are to be F, -measurable ensures 
that f(t) is adapted to the filtration F. The assumption that the n; are square 
integrable ensures that f(t) is square integrable for each t. Also, MjZ,, is a 
vector space, that is, af + bg € Måep for any f,g E€ Mžep and a,b E R. 


Definition 7.2 


The stochastic integral of a random step process f € Me of the form (7.2) is 
defined by | 


n—l1 
I(f) = $ n; (W(tj+1) — W(t;)). (7.3) 
j=0 
Proposition 7.1 


For any random step process f € Miep the stochastic integral I(f) is a square 
integrable random variable, i.e. I(f) € L*, such that 


B (IDP) =E( [7 OP at). 


Proof 


Let us denote the increment W(t;41) — W(t;) by A;W and tj+ı — t; by A;t 
for brevity. Then 


E (AW) =0 and E (44W) = Ajt. 


First, we shall compute the expectation of 


n—ln—l1 n—1 
IAP =$ So nine AjWARW = So AW +250 nym AW AW. 
j=0 k=0 j=0 k<j 


Since 7; and A;W are independent, 


E (n?A?W) = E (nå) E (44W) = E (nf) Ait. 
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If k <j, then njn,.A,W and A;W are independent, so 
E (njn AjW AkW) = E (njn AW) E(A;W) = 0. 
Therefore 


BOND I= D n?) 


It follows that I(f) € L?, since no, m, ...,n-1 € L’. 
On the other hand, 


n—ln—i n—1 
2 
FO = ` > 5M Ltj tiga) (E) Lte tes) (4) i > 5 Lle; tj) (4) 
j=0 k=0 JjJ=0 


implying that 


B([ OPa) = SB (i) 4 t 
E (IDP) =e( f erat), 


This means that 


as required. O 


Exercise 7.2 


Verify that for any random step processes f,g € Mžep 
EDIO) =E( [ FOO at). 


Hint Try to adapt the proof of Proposition 7.1. Use a common partition 0 = to < 
tı <---<t, in which to represent both f and g in the form (7.2). 
Exercise 7.3 


Show that I : M2 


Zep © I? is a linear map, ie. for any f,g € M3, and any 
a,b ER 


I(af + 8g) =al(f) + BI(g). 


Hint As in Exercise 7.2, use a common partition 0 = to < tı <--: < tn in which to 
represent both f and g in the form (7.2). 
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The stochastic integral J(f) has been defined for any random step process 
f € Mjep. The next stage is to extend J to a larger class of processes by 
approximation. This larger class can be defined as follows. 


Definition 7.3 
We denote by M° the class of stochastic processes f(t),t > 0 such that 


E(f OPa) Se 


and there is a sequence fj, f2,... € M s of random step processes such that 
P se 2 = 
tim E| [ HO- fol at) =0 (7.4) 


In this case we shall say that the sequence of random step processes f1, fo,... 
approximates f in M°. 


Definition 7.4 
We call I(f) € L? the Itô stochastic integral (from 0 to oo) of f € M? if 


Jim E (IIF) — Ifa?) =0 (7.5) 


for any sequence f1, fo,... € Mass of random step processes that approximates 
f in M?, i.e. such that (7.4) is satisfied. We shall also write 


[ save 
in place of I(f). 


Proposition 7.2 


For any f € M? the stochastic integral I(f) € L? exists, is unique (as an 
element of L’, i.e. to within equality a.s.) and satisfies 


B (pr) =E f OPa) (7.6) 


Proof 


It will be convenient to write 


Ifl = (f OPa) and Illz = VEGA 
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_ for any f € M? and 7 € L?. These are norms! in M? and L?, respectively. 
Let fi, fo,...€ Ms gs be a sequence of random step processes approximat- 
ing f € M?, i.e. satisfying (7.4), which can be written as 


im If - Fall ay = 0. 


We claim that I(f1), I(f2),...is a Cauchy sequence in L’. Indeed, for any £ > 0 
there is an N such that ||f — fn|ly2 < § for all n > N. By Proposition 7.1 


Im) = I(fa)|lp2 = |[I(fm — fr\\lp2 

Il fm = fall m2 

IF- falim + IF = falla 
a 

9 + 9 =E 

for any m,n > N, which proves the claim. 

Because L? with the norm ||-||;2 is a complete space (in fact a Hilbert 
space), every Cauchy sequence in L? has a limit. It follows that I( f1), I(f2),..- 
has a limit in L? for any sequence fı, fo,... of random step processes approxi- 
mating f. It remains to show that the limit is the same for all such sequences. 
Suppose that fi, fo,... and g1,g2,... are two sequences of random step pro- 
cesses approximating f. Then the interlaced sequence f1, 91, f2, g2,... approxi- 
mates f too, so the sequence I( f1), I(g1), I( f2), [(g2),... has a limit in L?. But 
then all subsequences of the latter sequence, in particular, I(f1), 7(fo),... and 
T(g1), [(g2),... have the same limit, which we denote by I(f). We have shown 
that 


IA ait 


A 


lim ICF) - I(fa)llz2 = 0 


i.e. (7.5) holds for any sequence fı, f2,... of random step processes approxi- 
mating f. 
Finally, by Proposition 7.1 


“Wales =U falar 


for each n, since the fn are random step processes. By taking the limit as 
n — oo we obtain 


HA = WF lage - 
But this is equality (7.6). O 


1 To be precise, the norms are defined on classes of functions, respectively, from M? 
and L° determined by the relation of equality a.s. However, we shall follow the 
custom of identifying such classes with any of their members. 
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Exercise 7.4 


Show that for any f,g E€ M? 


E (I(f)I(9)) = E ( f tow at) 


Hint Write the left-hand side in terms of E (|I(f) + I(g)|’) and E (|I(f) — (9)|”), 
the right-hand side in terms of E Cha IFE) + g(t)|’ dt) and E Th IFE — g(t)|? dt) 
and then use (7.6). 


Having defined the It6 stochastic integral from 0 to oo, we are now in a 
position to consider stochastic integrals over any finite time interval [0,7]. 


Definition 7.5 


For any T > 0 we shall denote by MA the space of all stochastic processes 
f(t),t > 0 such that 


lor) f € M? 
The Itô stochastic integral (from 0 to T) of f € MŻ is defined by 
Ir(f) =I (lof) - (7.7) 
We shall also write n 
[ soa 
0 


in place of Ir(f). 


Exercise 7.5 


Show that each random step process f € Mj, belongs to M? for any t > 0 
and | 


t 
(f) = i f(s) dW(s) 
sa martingale. 


Hint The stochastic integral of a random step process f is given by the sum (7.3). 
What is the conditional expectation of the jth term of this sum given F, if s < t;? 
What is it when s > t;? 


The processes for which the stochastic integral exists have been defined 
as those that can be approximated by random step processes. However, it is 
10t always easy to check whether or not such an approximation exists. For 
yractical purposes it is important to have a straightforward sufficient condition 
‘ar a nrocess ta have a. stochastic integral. In calculus there is a well-known 
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- result of this kind: the Riemann integral exists for any continuous function. 
Here is a theorem of this kind for the It6 integral. 


Theorem 7.1 


Let f(t),t > 0 be a stochastic process with a.s. continuous paths adapted to 
the filtration F. Then 


1) f € M’, i.e. the It6 integral I(f) exists, whenever 
E ( J | f(t)? at) < 00; (7.8) 
0 


2) f € MA, i.e. the Itô integral I7(f) exists, whenever 


E s PON a oe. (7.9) 
0 


Proof 


1) Suppose that f(t),t > 0 is an adapted process with a.s. continuous paths. If 
(7.8) holds, then 


k 
n k < k4 1 — a 2 


otherwise, 


is a sequence of random step processes in Mj. Observe that for any k = 


| Oe) eee 
[toe 


by Jensen’s inequality. We claim that 


OQ 


lim IF — fat) dt=0 as. 


11-4 OO 0 


k+l 2 


I, Í fatl dt =n < J IF| dt a.s. (7.11) 








This will imply that 
im E( S 1O- AOP d) =0 
11-4 OO 0 
by the dominated convergence theorem and condition (7.8) because 
| f(t) — fn(t)|* dt < 2 | (IF)? + lfn(¢)/?) dt 
. Oo 
<af IOP dt. 
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The last inequality follows, since 
/ lfn(t)|° dt < j IFO dt a.s. 
0 0 


for any n, by taking the sum from k = 0 to oo in (7.11). 
To verify the claim observe that 


OO N fee) 
/ F(t) — fa(t)[2 dt = | f(t) — fa(t)|? dt + | F(t) — fa(t)|? dt 
0 0 N 
N ee) 
<f O-O] (IOP + lf?) ae 


y 2 a 2 
< f O-AOPdt+A] OPd as 
0 N-1 
The last inequality holds because 


f. Ifn(t) asf ise) at < | f(t)? dt a.s. 


N-1 


n 


for any n and N, by taking the sum from k = nN to œ in (7.11). The claim 
follows because e 
lim fA dt=0 as. 
N->oo N-1 


by (7.8) and 
N 


lim f(t) —fn(t)|/’dt =0 as. 


tt CO 0 


for any fixed N by the continuity of paths of f. 

The above means that the sequence fi, f2,... € Mi.) approximates f in 
the sense of Definition 7.3, so f € M?. 

2) If f satisfies (7.9) for some T > 0, then 1ljo,7)f satisfies (7.8). Since f is 
adapted and has a.s. continuous paths, ljo,7) f is also adapted and its paths are 
a.s. continuous, except perhaps at T. But the lack of continuity at the single 
point T does not affect the argument in 1), so ljoryf € M?. This in turn 
implies that f € M+#, completing the proof. O 


Exercise 7.6 


Show that the Wiener process W(t) belongs to MÊ for each T > 0. 


Hint Apply part 2) of Theorem 7.1. 


Exercise 7.7 
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. Hint Once again, apply part 2) of Theorem 7.1. 


The next theorem. which we shall state without proof, provides a character- 
ization of M? and Ms, i.e. a necessary and sufficient condition for a stochastic 
process f to belong to M? or MŽ. It involves the notion of a progressively 
measurable process. 


Definition 7.6 


A stochastic process f(t),t > 0 is called progressively measurable if for any 
t>0 
(s,w) > f(s,w) 


is a measurable function from [0,t] x N with the o-field B[0,t]xF to R. Here 
B[0,t]xF is the product o-field on (0, t] x 2, that is, the smallest o-field con- 
taining all sets of the form A x B, where A C [0,¢] is a Borel set and B E F. 


Theorem 7.2 


1) The space M? consists of all progressively measurable stochastic processes 


f(t),t > 0 such that 
E d ) OO. 
{i JAV dt) < 


2) The space M4 consists of all progressively measurable stochastic processes 


f(t),t > 0 such that 
d 2 
e( | lf (t)| i) < 00. 


7.2 Examples 


According to Exercise 7.6, the Wiener process W(t) belongs to M# for any 
T > 0. Therefore the stochastic integral in the next exercise exists. 


Exercise 7.8 


Verify the equality 


a 1 1 
J W (t) dW (t) = zW TY anl 
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by computing the stochastic integral from the definition, that is, by approxi- 
mating the integrand by random step functions. 


Hint It is convenient to use a partition of the interval [0, T] into n equal parts. The 
limit of the sums approximating the integral has been found in Exercise 7.1. 
Exercise 7.9 

Verify the equality 


T T 
/ tdW(t) = TW (T) — / W (t) dt, 
0 


0 


by computing the stochastic integral from the definition. (The integral on the 
right-hand side is understood as a Riemann integral defined pathwise, i.e. sep- 
arately for each w € RN.) 


Hint You may want to use the same partition of [0,7] into n equal parts as in 
Solution 7.8. The sums approximating the stochastic integral can be transformed 
with the aid of the identity 


c(b — a) = (db — ca) — b(d—c). 


Exercise 7.10 


Show that W(t)? belongs to Mż for each T > 0 and verify the equality 


T T 
1 
f W(t)dW(t) = SW (T) — J W(t) dt, 
0 0 
where the integral on the right-hand side is a Riemann integral. 


Hint As in the exercises above, it is convenient to use the partition of [0,7] into n 
equal parts. The identity 


a? (b - a) = 5 (Ù? — a?) -a (b-a)? 5 b- a) 


can be applied to transform the sums approximating the stochastic integral. You may 
also need the following identity: 


(a? —b’)” = (a—b)* +4 (a — b)? b+ 4 (a — b)? b. 


7.3 Properties of the Stochastic Integral 


The basic properties of the It6 integral are summarized in the theorem below. 
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Theorem 7.3 


The following properties hold for any f,g € M7, any a,8 € R, and any 0 < 
s<t: 

1) linearity 

t 


/ (af (r) + 89(r)) dW (r) =a | f(r) dW(r) + B f g(r) dW (r); 
0 0 0 


e( ) -E(f UOPa): 


3) martingale property 


2) isometry 





J “f(r) dW(r) 





E (f t dW (r) F = [ tawo) 





Proof 
1) If f and g belong to M/, then 19 4) f and 1jo,4)g belong to M?, so there are 
sequences f1, f2,... and 91, 92,... in Mj.) approximating ljo,t) f and lj 4)g. It 


follows that ljo) (af + 8g) can be approximated by af; + 8g91,af2+Pg2,.-..- 
By Exercise 7.3 
I (afn zg Bgn) = al( fn) + BI(gn) 
for each n. Taking the L? limit on both sides of this equality as n — oo, we 
obtain 
I (1po,t) (af + B9)) = aI (Lot) f) + BI(1 0,29), 

which proves 1). 

2) This follows by approximating 194) f by random step processes in Mžep 
and using Proposition 7.1. 

3) If f belongs to M?, then ljo) f belongs to M?. Let fi, fo,... be a se- 
quence of processes in Méep approximating lio 4) f. By Exercise 7.5 


E (I (110,t) fn) \Fs) =I (1jo,s) fn) | (7.12) 


for each n. By taking the L? limit of both sides of this equality as n > oo, we 
shall show that 

E (I (10, f) |Fs) = T (1po,s)f) > 
which is what needs to be proved. Indeed, observe that 1io,s) f1, 1jo,s) fo,... is a 


sequence in Mgep approximating ljo,s) f, SO 


I (1jo,s) fn) +] (lios) f) in L? as n > o. 
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similarly, lio,t) fi» Lio, f2,... is also a sequence in Ms approximating 1ljo,t) J; 
vhich implies that 


I (lio) fn) +f (lof) in L” as n — œo. 
Che lemma below implies that 
E (I (lp) fn) |F) > E (I (lof) |F) in L? asn > œ, 


ompleting the proof. O 


emma /.1 


f € and 1, &o,... are square integrable random variables such that €, — € in 
32 as n > œ, then 


E (E€n|G) > E (EIG) in L? as n > œ 


or any o-field G on N contained in F. 


roof 
3y Jensen’s inequality, see Theorem 2.2, 

|E (Elg) - E (EID)? = |B (Em — 19)? < E (lén - €7|9), 
vhich implies that 


E (1E (Enl9) = E (&I9)1°) 


e (2 (Ie <6) 


E (l&n — €|’) +0 


IA 


Ss n > oo. O 


In the next theorem we consider the stochastic integral IN f(s)dW (s) as a 
unction of the upper integration limit t. Similarly as for the Riemann integral, 
t is natural to ask if this is a continuous function of t. The answer to this 
{uestion involves the notion of a modification of a stochastic process. 


definition 7.7 


set E(t) and C(t) be stochastic processes defined for t € T, where T c R. We 
ay that the processes are modifications (or versions) of one another if 


PIIA — 7(*\} — 1 for all te T (712) 
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Remark 7.1 
If T Cc R is a countable set, then (7.13) is equivalent to the condition 
P {€(t) = C(t) for alt € T} = 1. 


However, this is not necessarily so if T is uncountable. 


The following result is stated without proof. 


Theorem 7.4 


Let f(s) be a process belonging to M? and let 


ce) = FAW) 


for every t > 0. Then there exists an adapted modification ¢(t) of €(t) with a.s. 
continuous paths. This modification is unique up to equality a.s. 


From now on we shall always identify I f(s)dW (s) with the adapted mod- 
ification having a.s. continuous paths. This convention works beautifully to- 
gether with Theorem 7.1 whenever there is a need to show that a stochastic 
integral can be used as the integrand of another stochastic integral, i.e. belongs 
to MŻ for T > 0. This is illustrated by the next exercise. 


Exercise 7.11 
Show that ; 
c(t) = | Wawi) 


belongs to MA for any T > 0. 


Hint By Theorem 7.4 €(t) can be identified with an adapted modification having a.s. 
continuous trajectories. Because of this, it suffices to verify that €(t) satisfies condition 
(7.9) of Theorem 7.1. 


7.4 Stochastic Differential and Itô Formula 


Any continuously differentiable function x(t) such that z(0) = 0 satisfies the © 
formulae 


T 
x(T)? = 2 | O; 
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(T)? = TEG dx(t), 


where dz(t) can simply be understood as a shorthand notation for z‘(t) dt, the 
integrals on the right-hand side being Riemann integrals. Similar formulae have 
been obtained in Exercises 7.8 and 7.10 for the Wiener process: 


WT)? 


[ dt + of W (t) dW (t), 


T T 
3 2 
W(T) 3 | W(t) aes | W (t)2dW (t). 


Here the stochastic integrals resemble the corresponding expressions for a 
smooth function z(t), but there are also the intriguing terms id dt and 


3 ee W (t) dt. The formulae for W (T)? and W(T)°® are examples of the much 
more general [tô formula, a crucial tool for transforming and computing 
stochastic integrals. Terms such as a dt and 3 i W (t) dt, which have no ana- 
logues in the classical calculus of smooth functions, are a feature inherent in 
the Itô formula and referred to as the Itô correction. The class of processes 
appearing in the It6 formula is defined as follows. 


Definition 7.8 


A stochastic process €(t),t > 0 is called an Jté process if it has a.s. continuous 
aths and can be represented as 


T T 
EIT) = €(0) + | a(t) dt + J b(t) dW (t) a.s., (7.14) 
0 (0) 


where b(t) is a process belonging to M# for all T > 0 and a(t) is a process 
\dapted to the filtration F; such that 


T 
f la(t)|dt<oo a.s. (7.15) 
0 


or all T > 0. The class of all adapted processes a(t) satisfying (7.15) for some 
l > 0 will be denoted by £2. 
For an It6 process € it is customary to write (7.14) as 
d&(t) = a(t) dt + b(t) dW (t) (7.16) 


ind to call df(t) the stochastic differential of €(t). This is known as the Ité 
lifferential notation. It should be emphasized that the stochastic differential 
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understood in the context of the rigorous equation (7.14). The Ito differential 
notation is an efficient way of writing this equation, rather than an attempt to 
give a precise mathematical meaning to the stochastic differential. 


Example 7.1 
The Wiener process W (t) satisfies 


W(T) = J i dW (t). 


(The right-hand side is the stochastic integral I(f) of the random step process 
f = 1o,r).) This is an equation of the form (7.14) with a(t) = 0 and b(t) = 1, 
which belong, respectively, to L} and M?# for any T > 0. It follows that the 
Wiener process is an [tô process. 


Example 7.2 


Every process of the form 


. T 
(T) = &(0) + J a(t) dt, 


where a(t) is a process belonging to £L} for any T > 0, is an Itô process. In 
particular, every deterministic process of this form, where a(t) is a deterministic 
integrable function, is an It6 process. 


Example 7.3 


Since a(t) = 1 and b(t) = 2W (t) belong, respectively, to the classes £} and 
Mż for each T > 0, | 


W(T)? = i i dt+2 J i W (t) dW (t) 


is an It6 process; see Exercise 7.8. The last equation can also be written as 
d(W(t)*) = dt + 2 W (t) dW (t), 


providing a formula for the stochastic differential d (W (t)?°) of W(t)’. 


Exercise 7.12 


Show that W(t)? is an Itô process and find a formula for the stochastic differ- 
ential d(W(t)?). 
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Hint Refer to Exercise 7.10. 


Exercise 7.13 


Show that tW (t) is an It6 process and find a formula for the stochastic differ- 
ential d (tW (t)). 


Hint Use Exercise 7.9. 


The above examples and exercises are particular cases of an extremely im- 
portant general formula for transforming stochastic differentials established by 
Itô. To begin with, we shall state and prove a simplified version of the formula, 
followed by the general theorem. The proof of the simplified version captures 
the essential ingredients of the somewhat tedious general argument, which will 
be omitted. In fact, many of the essential ingredients of the proof are already 
present in the examples and exercises considered above. 


Theorem 7.5 (It6 formula, simplified version) 


Suppose that F(t,z) is a real-valued function with continuous partial deriva- 
tives Fy(t,z), F; (t,x) and Fý (t,x) for all t > 0 and x € R. We also assume 


T 


that the process F! (t, W (t)) belongs to M? for all T > 0. Then F(t, W(t)) is 
an Itô process such that 
: 


TN wie) | dt 


T 
F(T,W(T)) — F(0,W(0)) = Í (H.W) n 
ie / F'G, W(t) dW (t) as. (7.17) 


In differential notation this formula can be written as 


dF (t,W(t)) = (Faw) + zeat W(t) dt + F; (t, W (t))dW (t). (7.18) 
Remark 7.2 


Compare the latter with the chain rule 
dF(t,x(t)) = Fy(t,2(t)) dt + Fi (t, x(t)) dz(t). 


for a smooth function z(t), where dx(t) is understood as a shorthand notation 
for x'(t)dt. The additional term $F! (t, W (t)) dt in (7.18) is called the Itô 


correction. 
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Proof 


First we shall prove the Ito formula under the assumption that F and the 
partial derivatives F? and F? are bounded by some C > 0. 

Consider a partition 0 = tọ < t] < < t? = T, where t? = L of 
[0,T] into n equal parts. We shall denote the increments W(t?,,) — W(t?) 
by APW and t?,, — tp by Apt. We shall also write W? instead of W(t?) for 


brevity. According to the Taylor formula, there is a point W; in each interval 
[W (t?), W(t?,,)] and a point tẹ in each interval [t?, t?,,] such that 


F(T, W(T)) — F(0, W (0)) oe (thi WE) — Ft? WP) 


7=0 
n—l 
= (F ti, Wi+1) — F(t it1) ) + G F(t; W Fit pW) 
1=0 
n—1 n—l je 1 _ 
= 2 Fel (EWEA t+ So F(t}, WP)ArWw + = Do (t?, Wr) (AW) 
=0 1=0 2 
n—1 n—1 
= So Fi, Wr )Arte+ = 5 Fl to WryArte+ X Fi (t?, WP) AFW 
1=0 1=0 


ee a3 F" (te, WP) ((arw)? : Art) 


J p LF ( (t? WP) - Fin (t?, WP)| (A W}. 


We shall deal separately with each sum in the last expression, splitting the 
proof into several steps. 
Step 1. We claim that 


T 
lim PEG n )A?t= | F'(t,W(t)) dt as. 
0 


This is because the paths of W (t) are a.s. continuous, and F; (t, x) is continuous 
as a function of two variables by assumption. Indeed, every continuous path 
of the Wiener process is bounded on [0,7], i.e. there is an M > 0, which may 
depend on the path, such that 


|W (t)| <M for all t € [0,7]. 


As a continuous function, F;(t,z) is uniformly continuous on the compact set — 
(0, 7] x [—M, M] and W is uniformly continuous on [0,7]. It follows that 


lim sup |F; (EF, 1) — Filt, W (t)) | =0 as., 
n — CoO i.t 
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where the supremum is taken over alli = 0,...,n — 1 and t E [t?,t?,,]. By the 
definition of the Riemann integral this proves the claim. 
Step 2. This is very similar to Step 1. By continuity 


n— oo 1 


lim sup |F; (E,W) — Fi. (t,W(t))|=0 a.s., 


where the supremum is taken over alli = 0,...,n — 1 and t € [t?, t? ,]. By the 
definition of the Riemann integral 


T 
1i n n n 1 
wr F(t; WI) A; n= F, (t, W(t)) dt a.s. 


i=0 


Step 3. We shall verify that 
T 
im ŞO F(t? WP) AtW = / F'(t,W(t))dW(t) in L?. 


If Fi(t,z) is bounded by C > 0, then f(t) = Fi(t,W(t)) belongs to M?, by 
Theorem 7.1, and the sequence of random step processes 


In = >. I; (t7, Wi") liep en, .) E Mirip 
approximates f. Indeed, by continuity 
lim |fn(t) — f(t)|’ =0 foranyteé([0,T], as. 


Because |f,(t) — f (t)|? < 4C7, it follows that 
T 
ica i iG =FOC =O: as. 
noo 0 


by Lebesgue’s dominated convergence theorem. But So \fa(t) — HO dt < 


4TC?, so 7 
T 
lm E í [tao -sO at) =0 
n= o0 0 | 


again by Lebesgue’s dominated convergence theorem. This shows that f» ap- 
proximates f, which in turn implies that J(f,) tends to I(f) in L*, concluding 
Step 3. 
Step 4. If Fy! is bounded by C > 0, then 
n—1 
1 n n n 2 n . 
din, e ((Apw)? - Apt) =0 in L’, 
= 
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since 


ely F" (te, W?) ) ((apw) 2 Art) 
1—1 

= p2 E 
n—i 

= YD EFi (Wp)? E |(4pW)? - Are l 


1=0 








F” (t2, W2) (arw) a Art) Í 








n—l n—1 
<C YE kapw) sAr? 20 No (Ape? 


i=0 i=0 





T? 
os = 2 +0 as n — oo. 
1=0 


The first equality above holds because for any i < j 
E |F, (te, WP) (arw)? 5 Art) F! (t2, WP) ( (Anw)? — At) 
= E [FY (2, WP) ((arw)? - Art) Fu,(t7,Wp)| B[(Atw)? — are] 
= 0. 


This is because the expressions in the last two square brackets are independent 
and the last expectation is equal to zero. 
Step 5. By a similar continuity argument as in Steps 1 and 2 


Fy (t?,W?) — Fro(t?, W?)|=0 as., 





lim sup 
n=00 į 
where the supremum is taken over all i = 0,1,...,n—1. Since D (AtW)? > 


T in L? as n —> œ, there is a subsequence nı < ng < ... such that 


Nk—I1 
` (S*W) >T as. 


1=0 


as k > œ. It follows that 








Nk—l 
J (Falip Wet) — Falt, WP) ) (AP WY 
i=0 | 
n np—l 
pa (tg, With) — Fra lti”, With) > (AMW)* +0 as. 
1=0 








as k > oo. 
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In those steps above where L? convergence was obtained, we also have 


convergence a.s. by taking a subsequence. This proves the It6 formula (7.17) 


under the assumption that the partial derivatives F/(t,2) and F,(t,xz) are 


bounded. To complete the proof we need to remove this assumption. Let F(t, x) 
be an arbitrary function satisfying the conditions of Theorem 7.5. For each 
positive integer n take a smooth function Yn from R to (0, 1] such that yn(z) = 
1 for any z € [—n,n] and yp (xz) = 0 for any x ¢ [—n — 1,n + 1]. Then 


Fn(t, £) = pn(z) F(t, x) 


also satisfies the conditions of Theorem 7.5 and has bounded partial derivatives 
(Fa) (t,£) and (Fn)za(t, £) for each n. Therefore, by the first part of the proof 


F(T, W(T)) ~ Fa (0, W (0)) 
T 1 T 

= [ (EEWO) + jose) ) det | (ELEWO awa) 
0 0 


Consider the expanding sequence of events 
Ay = sup |W(t)| < n} l 
tE[0,T] 


Since F(t,z) = F,(t.x) for every t € [0,7] and z € [—n,n], it follows that 
(7.17) holds on A,. It. remains to show that 
Jim: P(A,) =1 


to prove that (7.17) holds a.s. But the latter is true because of Doob’s maximal 
L? inequality, Theorem 6.7, which implies that 


n° (1— P(An)) = P| sup |W(t)| > n} 
t€[0,T] 


2 
e( sup wo) 
tE[0,T] 
4E|W(T)|° = 4T, 


IA 


IA 


completing the proof. O 


Example 7.4 


For F(t,x) = x? we have F}/(t,x) = 0, F! (t,x) = 2x and F" (t,x) = 2. The Ité 
‘ormula gives 
d(W(t)*) = dt + 2W (t) dW (t), 


which is the same eaualitv as in Exercise 7.8. 
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Example 7.5 
For F(t,r) = z? we have F! (t,x) = 0, F! (t,x) = 3x? and Fi (t, z£) = 6x. By 
the Ito formula we obtain the same equality 


d(W(t)?) = 3W (t) dt + 3W (t)? dW (t) 


as in Exercise 7.10. 


Exercise 7.14 (exponential martingale) 


t 


Show that the exponential martingale X(t) = eW(}e77 is an Itô process and 


verify that it satisfies the equation 


dX(t) = X(t) dW (t). 


Hint Use the Itô formula with F(t, x) = ee” 5, 


As compared with the simplified version just proved, in the general Itô 
formula below W (t) will be replaced by an arbitrary Itô process €(t) such that 


dé(t) = a(t) dt + b(t) dW (t), (7.19) 


where a belongs to Lt and b to M? for all t > 0. In the general case the proof 
will be omitted. 


Theorem 7.6 (It6 formula, general case) 


Let €(t) be an It6 process as above. Suppose that F(t, x) is a real-valued func- 
tion with continuous partial derivatives F; (t,x), F} (t,x) and F} (t,x) for all 
t > 0 and x € R. We also assume that the process b(t) F} (t, €(t)) belongs to 
M+. for all T > 0. Then F(t, €(t)) is an It6 process such that 
| 1 
dF(t,€(t)) = (Fie) + Fi(t,€(t)) a(t) + 5 Fit (t,€(t)) u(t? | dt 


+ F' (t, €(t)) b(t) dW (t). (7.20) 


A convenient way to remember the It6 formula is to write down the Taylor 
expansion for F(t,z) up to the terms with partial derivatives of order two, 
substituting E(t) for x and the expression on the right-hand side of (7.19) for 
d&(t), and using the so-called [t6 multiplication table | 


dt dt = 0, dt dW (t) = 0, 
dW(t)dt=0, dW (t) dW (t) = dt. 
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his informal procedure gives 


1 1 
F, dt + F; dE + 5 ts dt dt + Fy, dt dé + 5 Fax dé dé 


F; dt + F; (adt+bdW) 


+E% dtdt + Fy. dt (adt + bdW) + = (adt + bdW) (adt + bdW) 


IE 


= Fidt+F, (adt+bdW)+ Fb dt 


l 
= (z: + Fiat zb) dt + Fb dW, 


hich is the expression in (7.20). Here we have omitted the arguments (t, €(t)) 


id, respectively, (t) in all functions for brevity. 


xercise 7.15 
pplying the Ito formula to F'(t,z) = 2", show that 


n _ a 


dW (t) ; ) W (t)"~* dt +nW(t)”* dW (t) (7.21) 


int This is a direct application of the It6 formula, but be careful with the assump- 
ons, in particular make sure that nW(t)”~' belongs to MŽ for all T > 0. 


xercise 7.16 (Ornstein—-Uhlenbeck process) 


uppose that a > 0 and ø € R are fixed. Define Y (t),t > 0 to be an adapted 
iodification of the Ito integral 


t 
YQ = vent | e°% dW (s) 
0 


ith a.s. continuous paths. Show that Y(t) satisfies 


dY (t) = —aY (t) dt + o dW (t) 


int Y(t) = F(t, E(t)) with E(t) = of e° dW (s) and F(t, x) =e7°®%r. 


"5 Stochastic Differential Equations 


‘his section will be devoted to stochastic differential equations of the form 


d&(t) = f(E(t)) dt + g(E(t)) dW (t). 
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Solutions will be sought in the class of Itô processes €(t) with a.s. continuous 
paths. As in the theory of ordinary differential equations, we need to specify 
an initial condition 


Here £o can be a fixed real number or, in general, a random variable. Being an 
Ito process, E(t) must be adapted to the filtration F, of W(t), so & must be 
Fo-measurable. 


Example 7.6 
The stochastic differential equation 
dX (t) = X(t) dW(t) (7.22) 


was used as a motivation for developing It6 stochastic calculus at the beginning 
of the present chapter. In Exercise 7.14 it was verified that the exponential 


martingale 
X(t) = ei Mens 


satisfies (7.22). It also satisfies the initial condition X(0) = 1. This is an ex- 
ample of a linear stochastic differential equation. For the solution of a general 
equation of this type with an arbitrary initial condition, see Exercise 7.20. 


Example 7.7 


In Exercise 7.16 it was shown that the Ornstein—Uhlenbeck process 
t 
Y(t) = vent | e** dW (s) 
0 
satisfies the stochastic differential equation 
dY (t) = —aY(t) dt + o dW (t) 


with initial condition Y (0) = 0. This is an example of an inhomogeneous lin- 
ear stochastic differential equation. See Exercise 7.17 for a solution with an 
arbitrary initial condition. 


Definition 7.9 


An Ito process €(t), t > 0 is called a solution of the initial value problem 


d&(t) = f(&(t)) dt + g(€(t)) dW (t), 
E(0) = £o 
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if & is an Fo-measurable random variable, the processes f(&(t)) and g(€(¢)) 
belong, respectively, to C} and M#, and 


T T 
EIT) = & + J f(€(t)) dt + J g(&(t))dW(t) as. (7.23) 


for all T > 0. 


Remark 7.3 


In view of this definition, the notion of a stochastic differential equation is 
a fiction. In fact, only stochastic integral equations of the form (7.23) have a 
rigorous mathematical meaning. However, it proves convenient to use stochastic 
differentials informally and talk of stochastic differential equations to draw on 
the analogy with ordinary differential equations. This analogy will be employed 
to solve some stochastic differential equations later on in this section. 


The existence and uniqueness theorem below resembles that in the theory 
of ordinary differential equations, where it is also crucial for the right-hand side 
of the equation to be Lipschitz continuous as a function of the solution. 


Theorem 7.7 
Suppose that f and g are Lipschitz continuous functions from R to R, i.e. there 
is a constant C > 0 such that for any z,y € R 

f(z) -— f) < Clz—yl, 

lg(z) - g(y)| < C |z — yl. 


Moreover, let £o be an Fo-measurable square integrable random variable. Then 
the initial value problem 


dE(t) = f(ECt)) dt + g(E(t)) dW (t), (7.24) 
E(0) = £o (7.25) 


has a solution (t), t > 0 in the class of Itô processes. The solution is unique in 
the sense that if ņn(t),t > 0 is another Itô process satisfying (7.24) and (7.25), 
then the two processes are identical a.s., that is, 


P {€(t) = n(t) for all t > 0} = 1. 
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Proof (outline) 


Let us fix T > 0. We are looking for a process € € MŻ such that 


S 8 
é(s) = & + J (E(t) dt + Í gE W(t) a.s. (7.26) 
for all s € [0,T]. Once we have shown that such a € € MŻ exists, to obtain 
a solution to the stochastic differential equation (7.24) with initial condition 
(7.25) it suffices to take a modification of € with a.s. continuous paths, which 
exists by Theorem 7.4. 
To show that a solution to the stochastic integral equation (7.26) exists we 
shall employ the Banach fixed point theorem in M2 with the norm 


T 
Il? = £ J ee (t)/? dt, (7.27) 


which turns M2 into a complete normed vector space. The number A > 0 
should be chosen large enough, see below. To apply the fixed point theorem 
define $ : M?, + M2. by 


BO) =t | FED f Ey awe (7.28) 
0 0 
for any € € M2 and s € [0,T]. We claim that ¢ is a strict contraction, i.e. 
P(E) -SOl < alle — Cla (7.29) 


for some a < 1 and all £,¢ € M4. Then, by the Banach theorem, @ has a 
unique fixed point € = (€). This is the desired solution to (7.26). 

It remains to verify that $ is indeed a strict contraction. It suffices to show 
that the two maps #, and $2, where 


5,(6)(s) = f HEE) dt,  BE)(s) = Í IEE) dW (t), 


are strict contractions with contracting constants a; and a2 such that a; +a2 < 
1. For $, this follows from the Lipschitz continuity of f. For $a we need to use 
the Lipschitz continuity of g and the isometry property of the It6 integral. Let 
us mention just one essential step in the latter case. For any £,¢ € M4 


T 
E | ers 
0 


T 8 
— ÀS _ 2 S 
E J i J ICE) — g(C(t))|? dtd 


2 
ds 





Í CEE) — 9(C(t))] awe) 





II82 (€) — BOI 


T 8 
œE f eo | Et) — CHI? dtds 


lA 
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T T 
2 —XAXs_Xt — Àt 2 d 
CE | ( Í ee is) e EE — C(t)? dt 


C? T = 2 C 2 
<E f eM ECE) — CCE)? dt = — 1E - CIR > 


Il 


lA 


ince f  e7>e ds = 4(1-e-(T-*)) < 1. Here C is the Lipschitz constant 
f g. If A > C*/e, then $z is a strict contraction with contracting constant < €. 

There remain some technical points to be settled, but the main idea of the 
roof is shown above. O 


-xercise 7.17 
‘ind a solution of the stochastic differential equation 
dX(t) = —aX(t)dt+adWi(t) 


rith initial condition X (0) = zo, where zo is an arbitrary real number. Show 
hat the solution is unique. 


lint Use the substitution Y(t) = e%* X(t). 
A linear stochastic differential equation has the general form 
dX(t) = aX (t) dt + bX (t) dW(t), (7.30) 


rhere a and b are real numbers. In particular, for a = 0 and b = 1 we obtain the 
tochastic differential equation dX (t) = X(t) dW (t) in Example 7.6. The solu- 
ion to the initial value problem for any linear stochastic differential equation 
an be found by exploiting the analogy with ordinary differential equations, as 
resented in the exercises below. 


-xercise 7.18 


uppose that w(t), t > 0 is a deterministic real-valued function of class C! such 
hat w(0) = 0. Solve the ordinary differential equation 


dx(t) = ax(t) dt + br(t) dw(t), (7.31) 
rith initial condition z(0) = zo to find that 
a(t) = roe% ttv ®) (7.32) 


We write dw(t) in place of w’(t) dt to emphasize the analogy with stochastic 
ifferential equations.) 
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Hint The variables can be separated: 
dx(t) 
x(t) 


By analogy with the deterministic solution (7.32), let us consider a process 
defined by 





= (a+ bu’ (t)) dt. 


KUN ree) (7.33) 


for any t > 0, where W (t) is a Wiener process. 


Exercise 7.19 


Show that X (t) defined by (7.33) is a solution of the linear stochastic differential 
equation 


b? 
dX (t) = (« + =) X (t) dt + bX (t) dW (t), (7.34) 
with initial condition X (0) = Xo. 

Hint Use the Itô formula with F(t, x) = e*t, 


Exercise 7.20 


Show that the linear stochastic differential equation 
dX (t) = aX (t) dt + bX (t) dW(t) 
with initial condition X (0) = Xo has a unique solution given by 
X(t) = Xpela- se erow (t) 
Hint Apply the result of Exercise 7.19 with suitably redefined constants. 


Having solved the general linear stochastic differential equation (7.30), let 
us consider an example of a non-linear stochastic differential equation. Once 
again, we begin with a deterministic problem. 


Exercise 7.21 


Suppose that w(t), t > 0 is a deterministic real-valued function of class Ct such 
that w(0) = 0. Solve the ordinary differential equation 


dz(t) = ~/1+ a(t)? dt+ /1+4 x(t)? dw(t) 
with initial condition z(0) = Zo. 


Hint The variables in this differential equation can be separated. 
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ercise 7.22 


ow that the process defined by 
X(t) = sinh(C + t+ W(t)), 


ere W (t) is a Wiener process and C = sinh’ Xo, is a solution of the stochas- 
differential equation 


dX (t) = ( 1+ X(t)? + 3x0) dt + (v 1+ X(t?) dW (t) 
th initial condition X(0) = Xo. 


nt Use the It6 formula with F(t, x) = sinh(t + zx). 


We shall conclude this chapter with an example of a stochastic differen- 
] equation which does not satisfy the assumptions of Theorem 7.7. It turns 
t that the solution may fail to exist for all times t > 0. This is a familiar 
enomenon in ordinary differential equations. However, stochastic differential 
uations add a new effect, which does not even make sense in the deterministic 
se: the maximum time of existence of the solution, called the explosion time 
ay be a (non-constant) random variable, in fact a stopping time. 


cample 7.8 


msider the stochastic differential equation 
dX(t) = X(t)>dt + X(t} aW (t). 


en 
1 


XO = wA 


a solution, which can be verified, at least formally, by using the Itô formula 
th F(t,x) = +. The solution X(t) exists only up to the first hitting time 


L=2- 
7=inf {t>0: W(t) = 1} 





1is is the explosion time of X(t). Observe that 
lim X(t) = co. 
t AT 


rictly speaking, the Itô formula stated in Theorem 7.6 does not cover this 
se, since F'(t,z) = -+ has a singularity at = 1. Definition 7.9 does not 
ply either, as it requires the solution X(t) to be defined for all t > 0. Suitable 
tensions of the Ito formula and the definition of a solution are required to 
udy stochastic differential equations involving explosions. However, to prevent 
| explosion of this book, we -have to refer the interested reader to a further 


eee — S25 Aah ae eae US Pee) ee 


oo Samm OS Re em EI ALBEIT ae RECS Lahm 


(toncanetsAeeeaneicn saananira aonn 


ARE ASE a a ALA ATSOA matae CHM An RAN GANNAD aa eit a Saba EA ESSELTE Sin. Me TSC SRR sib OAAR te SRT AA UAT en LN A a AA i 


aiK an aiian vn ENEE l AEREE lt ye RTT A i a aAA SEa. An AEA 
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7.6 Solutions 


Solution 7.1 
Using the first identity in the hint we obtain 


| 


n 


(W (tF)? — W(tF)”) 


M 


EWE WED -WE = 5 


J=0 
iais 2 
=; (W (t341) — W(t})) 
j=0 
= WT) 7 : =) (W (tj+1) - Wep)”. 
j=0 


By Exercise 6.29 the limit is 


n—l1 J 


lim 32 W (t7) (W (t741) — Wt?)) = =W(T) — zT 


Similarly, the second identity in the hint enables us to write 


SO WE) (Wega) — WED) = EE (Wa)? - WE?) 
j=0 j=0 
LEE (Wega) -WE 


iI 
© 


j 


(2 
n\\2 
= WT)? +5 P (W(t?,,) — W). 
=0 
It follows that 


lim DLONI (W (t21) -W(t?)) = =W(T)? + 


Solution 7.2 


For any random step processes f,g € Mep there is a partition 0 = tọ < tı < 
.- < tn such that for any t > 0 


n—1 n—l 
f t) 5 Nilit; t;41) (É) and g(t) = D Ci lit; t;41) 4) 
j=0 j=0 


where 7; and Ç; are square integrable F;,-measurable random variables for each 
PEE i 1 n — 1. (If the two partitions in the formulae for f and g happen to 
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ie different, then it is always possible to find a common refinement of the two 


iartitions. ) 
As in the proof of Proposition 7.1, we denote the increment W (t;41)—W (t;) 


y A;W and tji ae tj by Age. Then 


n—-1ln--1 
I(f)i(g) = >. N nj AW AW 
J=0 k=:0 
n—1l 
= SoG law? + So Ce AW AW +Y Gm ASW AW, 
j=0 j<k j<k 


here, by independence, 
E (yG |A;Ww/) = E (nj) E (lawi?) = E (jj) Ajt 
nd 


E (nik AW A.W) 
E (Gnr A;W AW) 


E (nik AjW) E (AW) = 0 
E (Gn âW) E (A.W) = 0 


| 


or any 7 < k. It follows that 


n—l 


E(I(f)I(g)) = $ E (ng) Ast. 
j=0 
‘herefore, it suffices to show that 


n—1 


E([ saat) = S E G) Ast 


ut this is true because 


n-in-l 


fltjg(t) = 5 > Nj Ck Lie; tigi) (E) Lex tea 1) (4) 
7=0 k=0 


n—l1 


z > 1569 Ue; ,t541) (t) 


j=0 


solution 7.3 


Ne shall use a partition 0 = tọ < tı <--- < tn such that 


n—1 n—i 
is >: Nglie;,tj41) and g= > Gi llt; t41)? 
=0 


j=0 j 
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where 7; and Ç; are square integrable F;,-measurable random variables for each 
j = 0,1,...,n — 1. (If the two partitions in the formulae for f and g happen 
to be different, then it is always possible to find a common refinement of the 
two partitions.) The increments W(tj41) — W (tj) will be denoted by 4;W for 
brevity. Then 


n—l1 


af + 8g = S (ani + BG) llt;tipa) 
j=0 
and 
n—1l 
I (af + Bg) = $ (an; + BG) AW 
7=0 
n—1 n—i 
= a ` nj â; W + B 3. 6; A; W 
j=0 j=0 
= al(f) + BI(g). 


Solution 7.4 


Consider the following scalar products in M? and L?: 


Toa =E i f(t)g(t) it) and = (7,C);2 = E (nQ) 


for any f,g € M? and n, € L*. They can be expressed in terms of the 
corresponding norms defined in the proof of Proposition 7.2, 


(odua = GI + ale — F f- Ilr 
(mOra = lin + Clie - F ln- Cle- 
Therefore Proposition 7.2 implies that 
(CG), I) = (F, 9) M2» 
which is the same as the equality to be proved. 


Solution 7.5 


If f € Mep is a random step process, then so is ljo} f € Mep C M? for any 
t > 0. This in turn implies that f € M? for any t > 0. 

We shall verify that (f) is a martingale with respect to the filtration Fi. 
Let 0 < s < t and suppose that f € Mep can be written in the form (7.2), 
where 


Vita <b << lras tea AeL la S l a L a: 
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ch a partition to,...,f, can always be obtained by adding the points s and 
f necessary. We shall denote the increment W(t,;41) — W (tj) by A;W as in 
2 proof of Proposition 7.1. Then 


m—l1 


lig f = >> gle, tj 
j=0 


m—1 
alf) =I(poyf) = X nj AW, 
j=0 


ich is adapted to F; and square integrable, and so integrable. It remains to 
npute 


m—l 


E(hi(f)|Fs) = E (Ilog) Fs) = X. E (nj; AjW|F,). 


j=0 
j < k, then 7; and 4;W are 7,-measurable and 
E (nj; A;W|F5) = nj A;W. 
j > k, then Fs C Fi, and 
E(njAjW|Fs) = E (E (n; AjW|Fi,) |Fs) 


E (jE (A;W|Fi;) |Fs) 
E (nj|\Fs) E (4jW) =0, 


ce n; is F;,-measurable and A; W is independent of F;,. It follows that 


k—1 
E(h(f)|Fs) = $ 19 AjW = I(ljo,sf) = Is(f). 


j=0 


lution 7.6 
definition, W (t) is adapted to the filtration F, and has a.s. continuous paths. 


yreover, 
T T | 

e( | |W E? a) = E (\w)/’) dt | 

0 0 : 


T 
= f tdt < oo. 
0 


Theorem 7.1 it follows that the Wiener process W belongs to M2. 


se E A ree cance paceman a 
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Solution 7.7 
Since W(t) is adapted to the filtration F;, so is W(t)”. Moreover, 


E ( [rear a = [ e (wor) dt 


T 
= J 3t? dt < oo. 


0 


Theorem 7.1 implies that W(t)? belongs to M2. 


Solution 7.8 
We fix T > 0 and put 


f(t) = lior (t)W (t). 
Then f € M? and 


[ W(t) dW (t) =[ 4 f(t) dW (t). 


Take 0 = tf < t? <- < t} = T, where t? = uw to be a partition of [0,7] 
into n equal parts, and put 


n—1 


f=, W (ti) Lier er, 4) 
i=0 


_ Then the sequence fi, f2,... E M2 step @Ppproximates f, since 
CO n—1 
2 n 
B(S HO-rOPa) =Y [os (IWO) - WEPIP) at 
i=0 “tr 


= khe (t — t?)dt 
: i 
= 9 ci tina — 


1=0 
1 Ty? 


==—-—0 asn-oo. 
2n 


By Exercise 7.1 
n n 1 2 1 
li) = 5 W(t) ) (W(t...) -W(tP)) > zW (T) — zT 


i=0 


_L? as n — co. We have found, therefore, that 


a 1 
J W (t)dW (t) = WT = 
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] 
-T. 
2 


olution 7.9 


et f(t) = t. Then lio rf belongs to M2. We shall use the same partition of 
T] into n equal parts as in Solution 7.8. The sequence 


= De lier 


»proximates lig 7) f, since 


e( f [Monst - rol it) 


follows that 


I(fa) > TW(T) - 


2 
_L? as n — œ. Indeed, by the classical inequality pare ai! <n Dr jail" 


„tr a) E MS 


step 


T 
e( f(t) — fn(t)| a) 


j Wi) 


n—1 biG 
=> f It — t?|* dt 
41 t7 
1 n—1 T3 | 
— 34+ 78 | 
(=l : 
T? : 
=——-+50 asnoo. : 
3n? | 
ith the aid of the identity in the hint, we can write the stochastic integral of ! 
, as | 
i) = 3 i? (W(t) — W(t?) | 
1=0 | 
oy a (t i1 W (tipa) — t? W (t?)) => W (t1) ( a t?) | 
i=0 i=0 | 
= )- W (t?,1) ( i441 7 ti). | 


id by the Cauchy-Schwartz inequality 


e([5 5 W (t1). 


i=0 


At a 





} 





N- f woe 
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n— | tipi i 
E | 2 (/ (W(t...) — W(t) a 
n— S 2 


Il 








IA 





an W(t%,4) — W(t) dt 





A 
aMi ji 


i+] — “i 
_ ea, —~=—-0 ano. 
= 2 = 2n? 2n 


This proves the equality in the exercise. 


Solution 7.10 
Using the same partition of [0,T] into n equal parts as in Solution 7.8 and 


putting 
= n\2 
fn = 5 W(t?) Liem tr, 1) 


1=0 
we obtain a sequence fi, fo,... € MŽ step Of random step processes which ap- 
proximates f = lio 7 W°. idee 


e([ lw - sto Pat) = D [re (IWA)? - wer)? )?) at 


1=0 


= i = (3 (t-t)? + 4(t-— er) dt 
i=0 YEr 


-¥|(2) +2(2) Z| 


3 
= — 30 anow. 
n 


The expectation above is computed with the aid of the following formula valid 
for any 0< s<t: 


E ( (w2 - W?)’) 


E ((Wi - Ws) ) + 4E ( (W; - W)? W) 
+4E (w: - W,} Ww?) 


3 (t — s)? + 4(t — s)s 
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‘sing the — in the hint, we can write 


I(fn) = p * (W (tÈ) — W(t?) 
1=0 
=> W (th)? -— W(t?)’) 
? l 
n—l1 
1 
-Ew ED (WR) = WE) -3 D (Weka) -wE 
1=0 
n-l 
= {W(r)' -X Wt?) (tha - tf) 
i=0 
n—i 3 
- P weer) [Wea -WE = (tha t?) 
i=0 
jo r a3 
-ESO (Wea) - ween)? 
i=0 
he L? limits of the last three sums are 
T 
lim S Wit?) (titr) = W (t) dt 
n—l 
. n n n 2 n n 
Jim, D Wen (WER) -WE)? - (a -t2)] = 0 
n—l 
ee 
Jim, © (Wa) = WE)? = 0 


Swe (2, - D- W (t) dt 


ideed, the first limit is correct because 
i=0 


AEL (W(t?) — W(t)) dt 
7 i E (IWE) - WHP) at 








K 








1==0 

= ` Ti (t — t?) dt 
1:20 t; 
T? 


= — > 0 asnoown. 
2N 
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) 
9 
= SE E (W(t?)*) _ = wit?))° — (t,1 — t?) 


1=0 


= D ti (ham 
7=0 


=a) he 
= e- S as n — oo. 


To the second limit can be verified as follows: 


E ( 5O W (E) [(W ERa) — WED)? = (tha — tp) 
1=0 
= Se (weer? | ween.) - WED)? = (a - 2) 


i=0 














) 





Finally, for the third limit we have 
2 


SE E (Wera) - weR))) 


n—1 


Y (WR) -we 


1=0 








i=0 
n—1 5 
= 06 2 (ti — t?) 
1=0 
n—1 
T: 6T? 
= 2 o ia as n œ> œ. 
‘= 


It follows that > 
Wis {Wry | W(t) dt 
| 0 


which proves the formula in the exercise. 


Solution 7.11 
We shall use part 2) of Theorem 7.1 to verify that 


= f W(s) dvs) 


belongs to M4 for any T > 0. By Theorem 7.4 €(t) can be identified with an 
adapted modification having a.s. continuous trajectories. It suffices to verify 


8 Basic Stochastic Processes 


at E(t) satisfies condition (7.9). Since the stochastic integral is an isometry, 


2 t t t2 
=E | IW (s)|° ds = sds = —. 
0 0 2 
follows that 


ef et) =E fw W (s) dW (s "Chee 


(s) dW (s) 














olution 7.12 


/e shall use the equality proved in Exercise 7.10: 
T 1 
W(T)? = 3 | W (t) dt+3 | W (t) dW (t). 
0 0 


he process 3W (t) belongs to £} for any T > 0 because it is adapted and 
AS a.s. continuous paths, so the integral J, 13W (t)| dt exists and is finite. 
y Exercise 7.7 the process 3W (t)? belongs to Mż for any T > 0. It follows 
iat W(t)’ is an Itô process. Moreover, the above equation can be written in 


ifferential form as 
dW (t)? = 3W (t) dt + 3W(t)* aW (t), 
hich gives a formula for the stochastic differential dW (t)°. 


olution 7.13 


. has been shown in Exercise 7.9 that 
T T 
W(T) = J W (t) dt + | t dW (t). 
0 0 


ince the Wiener process W(t) is adapted and has continuous paths, it belongs 
» £4, while the deterministic process f(t) = t belongs to MŽ for any T > 0. 
, follows that tW (t) is an It6 process with stochastic differential 


d (tW (t)) = W(t) dt + t dW (t). 


olution 7.14 


or F(t,x) = e*e~? the partial derivatives are F! (t, £) = —hete-2, Ptr) = 
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ete~2 and F" (t,x) = e"e~2. Since X(t) = e”(e-2, the Itô formula implies 


T 
that 


dX(t) = dF(t,W(t)) 
1 


(rie W (t)) + z zz(t, wi) ) dt + Fi (t, W (t)) dW (t) 


Il 


| 


(-5 x + 7x0) dt + X (t) dW (t) 
X(t) dW (t). 


| 


Because of this, to show that X(t) is an It6 process we need to verify that 
X(t) = eV (tea belongs to M2 for any T > 0. Clearly, it is an adapted 
process. It was computed in Solution 6.35 that HeW( = e2, SO 


T T , T 
E | xolda = | EeWOe-tdt = | dt =T < œ, 
0 0 


0 


which proves that X(t) belongs to M4. 


Solution 7.15 


Take F(t,x) = 2". Then F/(t,x) = 0, Fi(t,2z) = nz”™' and F! (t,£) = n(n — 
1)z"~*. The derivatives of F(t,z) are obviously continuous, so we only need 
to verify that F! (t, W (t)) = nW/(t)"—! belongs to Mż for T > 0. Clearly, it is 
adapted and has a.s. continuous paths. Moreover, 


T T T 
E | InW (t)i? dt = n? | E |W (H|? dt = | A2n—2t"* dt < œ, 
0 0 0 


where ap = 2/?q~1/2 (41) and P(x) = f.°t?~1e~* dt is the Euler gamma 
function. It follows by part 2) of Theorem 7.1 that Fi (t,W(t)) = nW(t)""! 
belongs to M2. Therefore we can apply the Itô formula to get 


d(W(t)") = mn Day (aya + nW (t)"-'dW(t), 
as required. 


Solution 7.16 


Some elementary calculus shows that F(t,z) = e~%'x has continuous partial 
derivatives such that F; (t,x) = —ae~'z, Fi(t,r) = e7% and Fi’ (t,£) = 0. 
Clearly, €(t) = o i e“ dW (s) is an It6 process with 


dE(t) = ce“ dW(t). 


Basic Stochastic Processes 


ce the function ae®’ F! (t,x) is bounded on each set of the form [0,7] x R, 
ollows immediately that oe®t F! (t, €(t)) belongs to M# for any T > 0. Asa 
sequence, we can use the It6 formula (the general case in Theorem 7.6) to 


dY (t) = d(e~*E(t)) 
= —ae~*€(t) dt + e~“* ae™ dW (t) 
= —aY (t) dt + o dW (t), 


ich proves that Y(t) satisfies the equality 
dY (t) = -aY (t) dt + o dW (t). 
lution 7.17 
ce F(t, x) = e*'x and consider the process 
Y(t) = F(t, X(t). 
an Y (0) = zo and 
dY (t) = dF(t, X(t)) 
(FIE X) - aX (FL, XE) + 507Fee(t,X(0)) dt 
+ oF! (t, X(t)) dW (8) 


(ae™ X(t) — ae™ X (t) ) dt + oe% dW (t) 
oe% dW (t). 


the Itô formula. It follows that 


Y(t) = zo + of e™ dW (s) 


X(t) = e Y(t) 
t 

e “x9 + oem | e** dW (s). 
0 


queness follows directly from the above argument, but Theorem 7.7 can 
» be used. Namely, the stochastic differential equation 


dX(t) = -aX (t) dt + o dW (t) 


f the form (7.24) with f(x) = —az and g(x) = o, which are Lipschitz 
tinuous functions. Therefore, the solution to the initial value problem must 
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Solution 7.18 


According to the theory of ordinary differential equations, (7.31) with initial 
= condition z(0) = zo has a unique solution. If zo = 0, then z(t) = O is the 
solution. If zo Æ 0, then 
x(t) 


In —— = at+ wit 
= w(t) 
by integrating the equation in the hint, which implies that 
x(t) = goet ttv), 


Solution 7.19 


By the Itô formula (verify the assumptions!) 
dX(t) = d (Koe owen) 


2 
Pn (axaert™" j: 5 xerttHwo ) dt + bX ett W (Ð aw (t) 
2 


b2 
(a + =) X (t) dt + bX (t) dW (t). 
This proves that X(t) satisfies the stochastic differential equation (7.34). As 


regards the initial condition, we have 


X (0) = Age ow) - = Xo. 


Solution 7.20 
The stochastic differential equation 

dX(t) = aX (t) dt + bX (t) dW(t) 
can be written as 


dX(t) = (c $ =) X(t) dt + bX (t) dW (t), 


where c = a — 2 By Exercise 7.19 the solution this stochastic differential 
equation with initial condition X (0) = Xo is 
X(t) = Xpet tow) 


mE (o- 4 )erow(e), 


The uniqueness of this solution follows immediately from Theorem 7.7. 


2 Basic Stochastic Processes 


ution 7.21 


e can write the ordinary differential equation to be solved in the form 


dx(t) 
1+ z(t)? 


= (1+ w'(t)) dt, 
uich implies that 
sinh’ z(t) — sinh 7} zo = t + w(t). 
ymposing the last formula with sinh, we obtain 
z(t) = sinh(c + t + w(t)), (7.35) 
yere c = sinh”! zo. 
ution 7.22 
nce F(t,z) = sinh(t + x) satisfies the assumptions of the It6 formula, 
dF (t,C + W(t)) 
(re C+W(t))+ Fil (t, C+ wo) dt 
+ Fi (t,C + W(t)) dW (t) 
(cosh(c +t+Wé(t)) + - sinh(C' + t + w) dt 
+ cosh(C + t+ W(t)) dW (t) 


(v 1 + sinh? (C + t+ W(t)) + : sinh(C + t + wi) dt 
+4/1+sinh*(C +t+W(t)) dW (t) 


(VIF XCF + 3x0) dt + (VIF XEF) aw (i). 


ne initial condition X (0) = sinh C = Xo is also satisfied. 
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